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ABSTRACT 



This document contains the final evaluation report of the 
Wisconsin Emerging Scholars Program 1993-94 and the script for the audio 
program. The evaluation report includes an executive summary; a discussion of 
the parameters of the evaluation including research questions and methods; 
implementation processes and outcomes for faculty and administrators; student 
learning processes and outcomes including those indicated by both qualitative 
and quantitative data; conclusions related to the use of the discussion 
section approach and the workshop approach; and recommendations related to 
pedagogical issues, out-of -class issues, and implementation issues. Interview 
guides, sample classroom observations, and a description of the Lead Center 
are also provided. The contents of the audio program include opening quotes 
which are excerpts from student interviews, an introduction to the program, 
excerpts from the student interviews of the comparison group, a brief summary 
of the outcomes of the pilot project, details of the three critical learning 
process factors, outcomes associated with the learning process factors, 
analysis of quantitative data, and a discussion of implementation problems. 
The learning process factors considered include group work, worksheets, and 
instructors. (DDR) 
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Executive Summary 



This report presents the evaluation findings of the pilot year (1993-94) of the Wisconsin Emerging 
Scholars (WES) Program, an “emerging scholars” program at the University of Wisconsin-Madison. 

It describes the WES Program in detail, explains the effect on students of the various academic and 
social features of the program, and indicates significant relationships between program features and 
their effects on students. The report also considers organizational factors that facilitated and inhibited 
the program leaders in their efforts to implement the program. 

The WES Program is designed for a group of first-year students that is not white male dominated and 
is equally academic and social in orientation. The 1993-94 WES cohort was comprised primarily of 
entering first-year students, half of whom were women, and 40 percent of whom were underrepre- 
sented ethnic minority students. This pilot group completed the first two semesters of calculus with 
grade point averages that were more than half a grade point higher than those of students in regular 
discussion sections. This was true even when measures of precollege mathematical and general 
mathematical ability were statistically controlled. The calculus achievement advantage of WES 
students generalized across sub-groups of various types, including women and ethnic minorities. 
Compared with students in the traditional discussion sections, the WES students also showed higher 
levels of confidence in their mathematical ability, greater comfort in performing calculus problems, 
learned to value multiple and creative ways of problem solving, and developed the interest and the 
ability to acquire a deeper, more conceptual understanding of calculus. 

The WES program was implemented at a major public university whose primary mission, as per- 
ceived by its faculty and staff, is research. In light of this context, there was moderate support for the 
progr am from all administrative levels — departmental, college, institutional and even extra- 
institutional — during the early implementation stages. This said, it is of note that the implementation 
process was sufficiently complex, time-consuming and, at times, precarious that leaders with less 
tenacity and vision might have abandoned the entire effort. Without the combination of formal and 
informal recognition, some financial support, and the personal commitment and tenacity of key 
leaders, it is unlikely the program would have been implemented. 

Program Description 

At UW-Madison students taking the calculus sequence (Math 221, 222, and 223) traditionally enroll 
in a large lecture which meets for 50 minutes three times a week and in a discussion section (DS) 
which meets twice a week for 50 minutes. Each of the three courses is worth five academic credits. 
Most lectures enroll 300 students each and are taught by a faculty member. Each discussion section 
seats approximately 20 students and is taught by a graduate teaching assistant (TA). The TA reviews 
the lecture and works problems on a blackboard in response to student requests. All students receive 
grades based predominantly on their performance on midterms and finals given in the large lecture. 
Only a small portion of a student’s grade is based on homework and discussion section participation. 
For each large-lecture midterm or final, all the exams are graded during a single session by the entire 
group of TAs, with one TA grading all 300 students on the same problem. This process ensures 
uniform grading and results in each TA having only a small part in grading his or her own students’ 
exams. 

Students in the WES program attend the large lecture, do regular homework problems, and are graded 
in the same fashion as everyone else. But instead of enrolling in a discussion section, these students 
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enroll in a “workshop,” for which they receive two “special topics” credits (graded pass/fail) in 
addition to five calculus credits. A workshop meets for two hours three times a week in a room 
reserved for the WES Program. In addition, WES students are encouraged to meet informally in this 
room whenever they wish, are invited to a few social events, and have the option of residing on the 
same dorm floor. In each work-shop session, the instructors give the WES students “worksheets” 
comprised of problems that are more difficult and more carefully designed than the homework 
assignments. Sitting around tables in groups of three or four, the WES students work the problems 
together, while the TA and student assistant roam the room asking strategic questions and offering 
hints when particular groups are obviously frustrated. The workshop instructors avoid directly 
answering students’ questions, and try to help students answer their questions themselves. In the 
1993-94 pilot, each of two WES workshops contained approximately 17 students, and was instructed 
by a faculty member acting in the TA role and an undergraduate “student assistant.” 

Why WES? 

Calculus persists as a serious barrier preventing ethnic minority, women and other students from 
entering mathematics-based disciplines (Hewitt & Seymour 1991, National Science Foundation 
1989). National data show that the calculus outcomes for students with comparable high-school 
GPA, college entrance, and college math placement scores vary by sex and ethnicity, with signifi- 
cantly smaller proportions of the women and ethnic minorities who enroll in calculus successfully 
completing the sequence. Because of the magnitude of this problem, it is important to understand not 
only if a calculus reform project such as WES is effective and efficient, but also why and how it does 
or does not work for various types of students. 

The “emerging scholars” calculus workshop approach has proved effective thus far in environments 
where highly charismatic people run the program and where the overall student population includes a 
relatively high proportion of ethnic minority students. The challenge open to the faculty WES 
committee is to demonstrate the effectiveness of the approach in a predominantly white, research- 
oriented university environment as that of the University of Wisconsin-Madison. 

Glossary of Special Terms and Acronyms 

Certain terms are used with precise meanings throughout the report. To clarify those meanings, the 
terms are defined here: 

Emerging Scholars program: 

The emerging scholars approach to learning calculus was developed at UC-Berkeley by Dr. Uri 
Treisman and further developed by the Dana Center at UT-Austin. It is intended as a type of 
honors program for underrepresented students and is based on the use of difficult worksheets and 
student-driven group work. 

formative evaluation: 

Evaluation is “formative” when applied researchers give faculty reformers feedback while the 
program/activity is being planned, piloted, and scaled up, and with the intention of improving 
the program/activity while it is under development. 
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outcomes: 

Traditionally, outcomes are the effects of a course or program which are measured in quantifi- 
able form. For example, grade point averages are a traditional outcome of a course. In this 
document, outcomes also include effects that are assessed in qualitative terms. For example, 
greater comfort with math or confidence in one’s mathematical ability are outcomes known not 
through analysis of database elements, but through analysis of extensive and complex interview 
material. 

DoIT: Division of Information Technology, the UW-Madison unit that, among other things, main- 
tains the institutions databases 

DS: discussion section 

MIC: minorities initiative committee 

SA: student assistant. For the 1993-94 pilot WES Program, the SAs were math majors and both had 
upper-division standing. 

UTIC: Undergraduate Teaching Improvement Council, a University of Wisconsin-System 
Administration resource 

WES: Wisconsin Emerging Scholars 

Use of Identifiers 

Except in the narrative portion of Section III, we refer to the large lectures and their associated 
sections with numbers. Thus, the faculty member who taught Large Lecture 1, and the faculty and 
graduate students who acted as instructors for the WES workshops and the discussion sections 
attached to Large Lecture 1 are called Lecturer 1, WES Instructor 1, and Discussion Section (DS) 
Instructor 1. We refer to the student assistants as SA-A and SA-B because these individuals alter- 
nated attending WES Workshops 1 and 2. 

Quoted Material 

Ellipses (...) in quoted material indicate deleted dialogue occurring within the reproduced material. 
Elipses are used so that readers can appreciate the speakers’ views on a particular topic without 
having to sort through the divergent twists and turns of the raw dialogue. The quoted material is 
presented as faithfully as possible to the speaker’s intent. Complete interviews (with identifiers 
removed) are available on request. Phrases added to provide clarity appear inside brackets [ ]. 

Interview dialogue is marked “I:” to indicate an interviewer’s speech, and is marked “L:” or “S:” to 
indicate speech of a lecturer or student. 
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I. The WES Evaluation Project: Who, What, Why, How 



A. Who 

This report is the final product of a 12-month (August 1993-August 1994) evaluation/ dissemination 
project funded by the NSF’s Division of Research, Evaluation, and Dissemination; the University of 
Maryland University College’s Institute for Research on Adults in Higher Education (IRAHE) under 
its Pilot Project on Efficiency in Learning; and the University of Wisconsin-Madison’s College of 
Letters and Sciences. The evaluation/dissemination project team was led by Susan B. Millar, who 
obtained the grants as a researcher with the Penn State Center for the Study of Higher Education and 
completed them in her new position as Director of UW-Madison’s new Learning through Evaluation, 
Adaptation and Dissemination (LEAD) Center. (Appendix C presents information on the LEAD 
Center.) The other members of the team were Baine B. Alexander, who began as an independent 
researcher and completed the work as the Associate Director of the LEAD Center, Professor Joel R. 
Levin, of the UW-Madison Department of Educational Psychology, and Heather A. Lewis, a graduate 
student in the UW-Madison Department of Mathematics who completed the grant as a researcher at 
the LEAD Center. Ms. Lewis’ work on this project was supported by the UW-Madison College of 
Letters and Science. 

B. What 

The evaluation/dissemination project combined “formative evaluation” with the development of 
dissemination strategies and materials. Evaluation is formative when applied researchers give faculty 
reformers feedback while the program/activity is being planned, piloted, and scaled up, and with the 
intention of improving the program/activity while it is under development. The evaluators assume 
that “effective strategies for improvement require an understanding of the process, a way of thinking 
that cannot be cap-tured in any list of steps or phases to be followed” (Fullan 1991, p. 67). Based on 
complex qualitative analyses supported by quantitative analysis, the evaluators sought to help pro- 
gram participants understand the organizational, cultural, and educational meanings of the change 
process in which they are engaged. In addition to helping program innovators understand and im- 
prove their programs, the evaluators intended that the systematic data on student outcomes and 
learning produced through the formal evaluation process would comprise material that program 
leaders could use for dissemination purposes. An audiotape dissemination program presenting 
analysis of the pilot program and segments of interviews with WES students, instructors and UW- 
Madison administrators is available from the UW-Madison LEAD Center. For additional informa- 
tion, interested readers also may contact the UW-Madison Department of Mathematics 
(608-263-3054) and ask for the WES Program coordinator. 

C. Why 

Education reform experts emphasize that the success of an innovative program depends, in part, on 
three key factors present at the time of program initiation: relevance, individual and organizational 
“readiness” or “capacity to use reform” (Firestone 1987), and resources (Fullan 1991, p. 63-64). 

The Wisconsin Emerging Scholars (WES) Program, a workshop-based calculus program that the 
University of Wisconsin-Madison offered for the first time in fall 1993, was promising in all three 
respects. 
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The WES Program is relevant because it seeks to increase the participation of under-represented 
students in the mathematics-based disciplines. 1 Designed to attract and serve predominantly ethnic 
minority, female and rural students, the WES Program is modeled on the “emerging scholars” pro- 
gram that Dr. Philip Uri Treisman pioneered at the University of Califomia-Berkeley in the early 
1980s and, with Effaim Armedariz and Jackie McCaffrey, developed further since the late 1980s at 
the University of Texas- Austin. It is important to note that the “emerging scholars” calculus work- 
shop model already has proved effective and efficient in particular environments. Although most of 
the 30 programs already in operation (Treisman 1991) have only anecdotal evidence of their effec- 
tiveness and efficiency, convincing evidence is available on the effectiveness of the calculus work- 
shop programs at UC-Berkeley, UT-Austin (Treisman 1991) and California State Polytechnic, 
Pomona (Cal Poly) (Bonsangue 1992, 1994). Nevertheless, many faculty and administrators at 
institutions like UW-Madison and Penn State suspect that the model “works” only under special 
conditions. In particular, they conjecture that the UC-Berkeley and UT-Austin programs are effective 
because of the efforts of an unusually charismatic, energetic, and committed person like Dr. Treisman 
and because these two institutions are located in states with high ethnic minority student populations. 
They venture that the Cal Poly program works because it is located in a comprehensive urban institu- 
tion whose primary mission is to serve the predominantly first-generation, and/or ethnic minority 
and/or returning adult students in its service area. In other words, the calculus workshop model has 
not yet been proven to be effective when located at an institution that is almost entirely white and 
primarily devoted to fulfilling a basic research mission, and when led by typical research mathemati- 
cians. Thus, if the Wisconsin Emerging Scholars program could produce convincing evidence that 
the emerging scholars approach works, educators in mathematics — -and other disciplines — would have 
confirmation that this model can be effective in predominantly white, research-oriented settings. 

In addition to having relevance, the WES Program displayed individual and organizational readiness, 
and had sufficient institutional resources. A detailed account of how the Program leaders helped 
create institutional readiness and marshall program resources is provided in Part III, “WES Process 
and Outcomes for Faculty and Administrators.” 

D. Research Questions and Methods 

The evaluation questions and research methods were based on the WES Program leaders’ objectives 
and formulated in light of the research literature on calculus reform efforts nationally. 

1. Learning Processes and Outcomes for Students 
a. Faculty Objectives 

During evaluation interviews (see below, “Methods for Gathering Evaluation Data”) we asked the 
WES instructors to formulate their objectives for the workshops. To help place the WES instructors’ 
course objectives into context, we also asked Lecturers 1 and 2 to describe their course objectives. 



1 Preliminary data indicate that calculus outcomes at UW-Madison conform to the pattern found elsewhere, showing 
that significantly smaller portions of women and ethnic minorities successfully complete the calculus sequence, 
compared to white and Asian males. 
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1) WES Instructors’ Objectives for Student Learning 

WES Instructors 1 and 2 each brought different commitments to their program. Instructor 1, a 
mathematician hired as an instructor, had taught college mathematics for several years, had experi- 
ence with outreach efforts, and recently had begun experimenting with different approaches to 
teaching. Instructor 2, a tenured professor who has been teaching at UW-Madison for more than 20 
years, had a long history of activist work in support of underrepresented people. WES Instructor 2 
also was the WES Program Director. 

The WES instructors described three key learning objectives for workshops. They wanted their 
students to: 

• learn the calculus at a sufficiently deep level that they would feel secure in the knowledge that 
they are not just “going through the motions,” plugging numbers into formulas; 

• cultivate a capacity to use multiple ways to solve problems; and 

• develop their ability to rely on themselves to formulate and solve problems. 

As Instructor 1 put it, “I do really think that the more you struggle with it yourself, the more 
powerful that feeling [of excitement and discovery] is. And so, yes, you can get it to some ex- 
tent when someone explains it well, but you get it at a much deeper level if you struggle with it 
yourself. So that would be my goal — to have as many of them get that at as deep a level as 

possible.” 

In more general terms, they also wanted the course to: 

• provide a math experience that attracts more students into math, science, and engineering majors; 

• provide informal academic and personal advising, and both planned and informal socializing 
opportunities to help first-year ethnic minority and rural students adjust to a research university 

environment. 

2) Lecturers’ Objectives for Student Learning 

Lecturer 1 and 2 had strongly contrasting levels of experience teaching large lectures in calculus. 
Lecturer 1, who was tenured relatively recently, was teaching the course for the first time. Lecturer 2 
had taught the course quite regularly for some 22 years. This difference in experience may explain, 
in part, the difference in their responses when asked about their objectives for the course. As indi- 
cated in the quotes below, Lecturer 1 had not felt encouraged to formulate specific course goals, and 
by default pursued the goals implicit in the Department’s existing syllabus. Lecturer 2, over his many 
years’ experience with the course, had formulated specific objectives: he wanted the students to learn 
how to translate an “ill-formulated” problem into mathematical terms, and see the relationships 
between the problem and the mathematical techniques presented in the lectures. He also indicated 
that another course objective was to “recognize and reward the very best students.” 

The following excerpt from an interview with Lecturer 1 presents his response to a question on 
course goals. 

I: What are the objectives you have for the course, in terms of the student learning, in terms of 

the objectives of the department, and for that matter in terms of the objectives of the course 
for other departments across the campus... How would you relate all those objectives to the 
way you structure the curriculum? Big question. 
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LI: Yeah, well, it has a very short answer. I have never thought about these things. I got an of- 
ficial letter asking me to apply here and, well, the interview that I got here and everything, 
the emphasis was on the research. And it was very clear to me that I was hired here because 
of various things that I have published and the talks that I’ve given, the people that I have 
talked to. And it did say something about “the department hopes that I will contribute to 
teaching here.” But there was never, um, before I got tenure the chairman came and sat in 
my class and observed and some teaching evaluations were done. I don’t think a lot of at- 
tention was paid to teaching... So, I consider myself a professional amateur when it comes to 
teaching. I had never really thought about these things.... 

I: Implicit in what you’re doing... are some implicit objectives or philosophies 

pertaining to what you were doing.... 

LI: I still don’t have clear objectives. What I basically do, the department has a syllabus, and a 
book that some committee has picked out. So I try to follow the book. 

The next interview excerpt presents Lecturer 2’s response to a similar question on course goals. 

I: Would you tell me about your perspectives on the objectives of the course for the students, 

for the department and for the other departments that calculus serves? It’s a big question. 

L2: What one will say, of course, is you want to teach the students to think and you want to teach 
the students to work hard. This means that ideally you want the students to take an ill- 
formulated problem or word problem and translate it into a mathematical formulation and 
see the relationship between the geometric or practical or the word problem and the tech- 
niques they’ve been studying. Now it’s very hard at this level even to get students to plug 
into a formula... I want to accomplish this objective in getting the students to think. I’m not 
particularly concerned that they should learn this or that, because no particular thing we 
teach seems to me to be very important, but the process, the whole process it seems to me is 
important... I think most of the professors are happy if you can just get the students to sort of 
understand what the notations mean and how to plug in the notations.... 

There is a feeling that we have to recognize and reward the very best students... An 
important part of the job is, you know — 40% of academia at least is evaluating other people, 
if not students, colleagues. It seems to be the main output of the university is to spend a lot 
of time evaluating other people. It’s unpleasant. Of course it’s not as unpleasant as being 
evaluated, but it’s still unpleasant to do the evaluating. It seems to be in the nature of the 
game. There are, of necessity, scarce resources. You have a great professor or a great 
mathematician, say, at Harvard. Now he can only have two or three students a year, so he 
has to pick the best. I mean, if you send him the weaker students you’re wasting both of 
their times, you’re not making the optimum use of the resource. So there has to be some 
kind of evaluation procedure, it’s in the nature of the game. It’s not a question, you couldn’t 
pour in more money and produce more brilliant people, because no matter how much money 
you pour in, right, there’s still going to be someone who’s the best and that’s the person 
who’s going to be in demand. 

While only two lecturers’ views on course goals are represented here, conversations we have had with 
many research mathematicians indicate that the goals articulated by Lecturer 1 and 2 are shared 
widely among math faculty with levels of experience comparable to these two people. 
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b. Existing Research 

Convincing evaluation research on calculus learning experiences is available. Of the data available 
on the workshop programs that Uri Treisman initiated and sustained at UC-Berkeley and UT-Austin 
(Culler 1982; Fullilove 1986; Fullilove and Treisman 1990; Asera 1990; and Treisman 1985, 1990, 
Myers & McCaffrey 1994), the piece by Fullilove and Treisman provides the most complete quanti- 
tative analysis of student outcomes, is readily available, and also provides a detailed program 
description. 

Data cited by these studies indicates striking success for emerging scholars programs. For example, 
adjusting for differences in incoming predictors of success* Fullilove and Treisman show that mathe- 
matics workshop students at UC-Berkeley were much more likely to earn grades of B-minus or better 
(taking 1981 as a characteristic year, 50% as opposed to 10% of non-program participants), much less 
likely to earn grades of D-plus or lower (taking 1981 as a characteristic year, 2% as opposed to 38%), 
and much more likely to persist and graduate (e.g., of African American students with incoming SAT 
math scores of between 550-800, 77% of program participants were retained or graduated as opposed 
to 49%). 

Similarly, Myers & McCaffrey report that at UT-Austin, 

“ ...fewer than one-third of African American and Hispanic American students who took 
calculus in the five years prior to the [Emerging Scholars] program’s inception earned 
grades of A or B. Only one half of those who were well prepared (as measured by a 
math SAT of 600 or above) earned such grades. In contrast, approximately 80 percent 
of the African American and Hispanic American students who have participated in 
ESP [Emerging Scholars Program] since its inception in 1988 have earned grades of 
A or B in calculus. ” 

Bonsangue posts similar outcome data for the workshop program at Cal Poly-Pomona. Researching 
133 workshop and 187 nonworkshop Black and Latino students who had no significant differences in 
mathematics background and achievement indicators, Bonsangue found that the 

“ workshop students achieved a mean grade of more than six-tenths of a grade point 
above nonworkshop students in first and second-year calculus (t=3.75, p < .001). 
Moreover, within three years after entering the institution, more than half (52%) of the 
minority nonworkshop students had either withdrawn from the institution or changed 
to a nonmathematics-based major, compared to fifteen percent of the workshop stu- 
dents.... Nonworkshop students required an average of one full quarter more to com- 
plete their three-quarter calculus sequence due to course failure. Individual records 
showed high patterns of course repeating, with nearly half (46%) of the nonworkshop 
minority students requiring five or more quarters to complete a three-quarter calculus 
sequence, compared to fewer than one-fifth (17%) of the workshop students. ” 
(Bonsangue 1994, p. 8) 

Supporting these quantitative data, the above-noted research (taken together) provides insightful 
analyses of the roles of the worksheets, student-instructor and student-student dynamics, and program 
development processes. In addition, drawing on his extensive experience with the programs at UC- 
Berkeley and UT-Austin and with scores of other workshop programs at diverse types of institutions, 
Treisman provides insightful analyses of factors critical to effective calculus learning (Treisman 
1992). 
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Faculty involved in other calculus reform activities also are reporting informal assessment data in 
video and symposia formats. For example, several calculus reform programs were featured in a 
nationally broadcast interactive video workshop sponsored by the California State University System 
in fall 1993. Informal findings also are reported by calculus reformers at professional meetings and 
at institutional symposia such as the speakers’ series sponsored by the UW-Madison College of 
Engineering, which featured the Mathematica Calculus Program at the University of Illinois at 
Champagne-Urbana (Uhl), the calculus program at Cornell University (West) and the Harvard 
calculus curriculum (Osgood). 

The informal evaluation data reported from these latter programs and the work produced 
by Treisman are weak for the same reason they are strong: they are produced by people who are 
deeply involved in making the programs work. Outsider researchers, such as Bonsangue and the 
WES evaluation team, can benefit from these scholars’ knowledge and experience by using their 
insights to help formulate research questions for use in their more formal studies. 

Research which investigates the fundamental processes of learning (for example, Lave 1988; Lave & 
Wenger 1991; Rogoff 1990; Vygotsky 1978; Wertsch 1991) and which describes the character and 
effects of cooperative learning environments at the college level (for example, Bruffee 1992, 1993; 
Conrad, Haworth, and Millar 1993; Gabelnick, Johnson, Johnson & Smith; MacGregor, Matthews, & 
Smith 1990; Goodsell, Maher, & Tinto 1992; Millar 1994; Svinicki 1990; Tobias 1992b) also is 
relevant to this evaluation project. These studies indicate why the learning processes used in WES 
workshop succeed, and substantiate that these processes work in diverse contexts as long as certain 
minimal conditions sustaining interaction focused on challenging problems are met. 

c. Research Questions 

While WES Instructors 1 and 2 formulated clear learning goals for the WES project, they did not 
posit particular relationships between these goals and planned workshop learning activities that they 
wanted evaluated. The evaluation team therefore formulated the following research questions in light 
of both existing evaluation research and the WES instructors’ goals. 

1. Overall, do WES students leant calculus better than DS students? More specifically, 

• What is the role of the worksheet problems in the learning experienced by WES students? 

• Do WES students develop their capacities to be resourceful problem-solvers who use multiple 
approaches? 

• Do WES students become more self-reliant thinkers? 

Rationale : Research cited above on other emerging scholars programs indicates that students 
leam calculus better when they are given problems that are more challenging than typical 
homework problems, cultivate a capacity to use multiple ways to solve problems, and develop 
their ability to rely on themselves to formulate and solve problems. 

2. Are the WES instructors (both SAs and workshop instructors) interacting with the students in 
ways that facilitate student learning, and if so, what are the key features of these interactions? 

Rationale : Informal classroom evaluation data and formal research on cooperative learning indi- 
cate that students are more able to gain “ownership” of calculus concepts when instructors act as 
guides-on-the-side rather than as lecturers. 
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3 . What roles do work groups play in this learning? What are the necessary characteristics of work 
groups that effectively foster learning? 

Rationale : Informal classroom evaluation data and formal research on cooperative learning sug- 
gest that students learn calculus better when they work problems in cooperative work groups 
than when they work alone. 

4. Are course retention and performance outcomes for WES students comparable to the outcomes 
achieved by students in other workshop programs, when analyzed by sex and ethnicity? Do 
WES students persist longer in science, mathematics, and engineering (SME) majors than 
non-WES students? 

Rationale : Data on other emerging scholars programs show that compared to students in non- 
workshop programs, students — and particularly women and underrepresented ethnic minority 
students— are retained in calculus at higher rates, earn higher grades when they participate in 
calculus workshops, and persist in SME fields at higher rates. 

d. Data Gathering Methods 

We pursued these research questions by collecting standard quantitative student data and by using 
various qualitative research methods. We collected multiple types of data so that we could triangulate 
across a range of different data sources during the analysis stage. In addition, we used a diversely 
trained research team comprised of two anthropologists, a math graduate student, an educational 
psychologist. Using researchers from different disciplines enhances the quality of the analysis; 
people trained in different “ways of knowing” see different things in the same data. 



The data collection methods used included: 

Interviews'. Our plan was to interview a total of 36 students, including nine students from the WES 
workshop and one discussion section attached to each of two large lecture classes, five times during 
the academic year. All 36 would participate in informal, audiotaped interviews. Three women and 
three men from each section would be interviewed in a group, and the remaining three from each 
section would be interviewed one-on-one. Interviews would be held at the beginning, midpoint, and 
end of the fall semester and at the midpoint and end of the spring semester. Students who dropped 
would be asked to participate in an “exit” interview. 
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Interviews actually occurred in the following configurations. (Due to scheduling difficulties, three 
spring 1994 WES focus groups were mixed sex groups.) 





Focus Grou 


Interviews 


Individual Interviews 




Female 


Male 


Female 


Male 


Fall 1993 


i 








WES 1 


3 


2 


2 . 1 


3 


WES 2 


4 


3 


1 


3 


DS 1 


4 


3 


1 


2 


DS2 


3 


2 


0 


2 


Spring 1994 










WES 1 


3 F 


1 M/1 F 


i 


i 


WES 2 


2 F / 1 M 


1 M/1 F 


2 


2 


DS 1 


— 


— 


1 


1 


DS 2 


— 


— 


1 


— 


DS 3 


3 


2 


1 


1 


DS 4 


3 


2 


— 


— 



The number of students in focus groups varied because some dropped the course or did not show for 
interviews. Also interviewed during the spring were one woman and one man who switched from 
221 WES to 222 DS. For Math 221, we selected the DS Sections taught by the TA coordinators, who 
were the most experienced of the approximately 10 TAs associated with each of Lecture 1 and 
Lecture 2. Both were white males. This choice is important because fall semester Math 221 discus- 
sion sections generally are taught by first-semester graduate students who have no prior experience 
teaching the course. We did not want to compare sections taught by inexperienced discussion section 
instructors with WES sections taught by very experienced WES instructors. For Math 222, we 
selected discussion sections taught by a female TA and by a white male who was not USA-born. To 
obtain enough DS students for interviews during the spring semester, we had to draw students from 
both of the sections taught by these two TAs. 

The interview protocols used for Math 221 appear in Appendix A. Similar protocols were used for 
Math 222. 

Free-writes: As planned, most study participants were asked to write a 15-minute “free- write” 
response to a question about his/her experiences in the calculus class. 

Collection of Course Artifacts'. The students who participated in one-on-one interviews were also 
asked to collect their course “artifacts” (handouts, homeworks, exams) for us. Of the 14 one-on-one 
students, we collected artifacts from eight. We photocopied these materials to assemble a passive 
record of the course from students’ points of view. 

Class Observations: Two evaluators observed each of the two WES and two Discussion Sections 
once in the middle and again toward the end of each semester. 

Video Records: Two meetings of the two WES and the two participating Discussion Sections were 
videotaped, all during the second half of the semester. 
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Student Database: End of term “snapshots” were obtained of the records of all students who 
enrolled in calculus from fall 1989 through spring 1994. Each student record included background 
data (gender, ethnicity, SAT and ACT scores, college math placement test data high school size and 
geographic location, high school rank and GPA, intended major, college classification, date o 
matriculation, and grades in college calculus courses). 



2. Implementation Processes and Outcomes for Faculty and Administrators 
a. Faculty and Administrators’ Objectives 

During interviews we asked the WES Program and other administrators to articulate their objectives 
for the program. These administrators include the WES Program Director, the chair of the Ma 
Department’s informal “minorities initiative committee” (MIC), the Math Department Chair, t e 
Dean of the College of Letters & Sciences, and the Associate Dean for Natural Sciences of the 

College of Letters & Sciences. 



The department chair and college-level administrators supported the idea of a WES Program because 
they believed (based on evidence from other emerging scholars programs) it could improve the 
calculus course outcomes for all students, and particularly for underrepresented ethnic minorities and 
women Simultaneously, they expressed serious doubts about whether such a program would be 
successful in the UW-Madison environment, and whether, if proven successful, the institution could 
afford what appear to be the higher instructional costs of workshop calculus. We therefore take as 
these individuals’ goal for the WES Program that it be shown to improve course performance out- 
comes, particularly for underrepresented students at UW-Madison, at a cost which the institution can 

support. 



While sharing this general program goal, the WES Program Director and MIC chair also e mor ® 
specific objectives for the Program. In addition to the more strictly academic goals articulated in the 
section on outcomes for students above, their goals included: 



• acquire program infrastructure and develop an administrative process that sustains high level 
Program performance without making demands on faculty beyond expected levels of service to 

the Department; . 

• pending strong academic outcomes, institutionalize and scale up the Program by persuading the 
Math Department and College of L & S to make WES workshops in calculus 221, 222, and 223, 
linear algebra, and differential equations available to underrepresented students, and possibly all 

students, who might prefer this learning approach; 

• obtain high quality course evaluation and program evaluation data, and develop dissemination 
materials for use across the UW-Madison campus and elsewhere. 
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b. Research Questions 

In light of faculty and administrators’ goals for institutional processes and outcomes and the research- 
ers’ knowledge of reform program processes, we posited the following questions to guide the evalua- 
tion component of the study: 

1. What progress, if any, has been made toward developing an infrastructure and an administrative 
process that is sustainable within the institutional reward structure? 

2. What are the prospects for institutionalization and scale-up of the WES Program? 

3. What are the results of the formative evaluation activity? 

4. What are the total instructional and administrative (including any evaluation and dissemination) 
costs of the WES Program? 

5. Are the costs of the program justifiable in terms of program benefits? 

c. Data Gathering Methods 

We pursued research questions by using the guided open-ended interviews of administrators and 
instructors and budget data provided by the Department Chair. The following individuals were 
interviewed: 

WES Instructors (twice per semester) 

informal chair of Math Department “minorities initiative committee” 

Department Chair (once per semester) 

L&S Associate Dean for the Natural Sciences (spring semester) 

Dean of College of L&S (once per semester) 

In addition, frequent informal conversations were held with two individuals who are attuned to 
administrative activity at both college and central levels: the Associate Dean for Physical Sciences of 
the Graduate School, and the Director of a major engineering education grant which contributed 
support to the WES Program as of summer 1994. Interviews focused on each individual’s objectives 
for WES and the organizational and structural problems and successes they experienced while trying 
to achieve these objectives. 

To provide real-time formative feedback, the WES evaluation team periodically provided WES 
instructors with informal briefings and an Interim Report (March 1993). 
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II. Learning Processes and Outcomes for Students 



Part II presents research findings in three different ways, in each case first highlighting the discussion 
sections and then the WES workshop. We begin (Section A) by describing what would be evident to 
a person who was passively observing the activities associated with the program: we describe typical 
activity in both discussion sections (DS) and WES workshops. In Section B we focus on student 
learning processes, first featuring the student points of view. Quoting from transcribed interviews, 
we convey themes that students characteristically expressed. Still relying heavily on quoted material, 
we then present our own analytic generalizations about how the students experienced the DS and 
WES learning processes. Section C treats student learning outcomes, first as measured by quantified 
indicators of achievement and course grades, and second as indicated by qualitative data gathered 
during interviews and observations. 

A. From an Observer’s Point of View: Key Features of the WES Program 



1. The Lectures 

Both DS and WES workshop students attended three 50-minute lectures each week of the fall (221) 
and spring (222) calculus courses. These lectures were given in large amphi-theater classrooms 
furnished with seats, each with a retractable board attached on its back for use by the occupant of the 
seat behind it. The lecturer was provided with a stage area containing a desk, podium and large 
expanse of blackboard space. Students generally remained with the same lecturer for both 221 and 

222 . 

Two weeks into the fall course, Lecturers 1 and 2 had taught 306 and 328 students, respectively. At 
the end of the term they were teaching 253 (50 drops, 3 withdraws) and 256 (66 drops, 6 wit raws) 
students respectively. Two weeks into the spring course, these Lecturers taught 175 and 273 students 
respectively, and by the end of the term taught 128 (43 drops, 4 withdraws) and 239 (29 drops, 5 
withdraws) respectively. (Historical data show that, of fall semester students, approximately 15 /o 
drop and 2% withdraw from Math 221 and approximately 16% drop and 2% withdraw from Math 

222 .) 

Lecturers 1 and 2 scheduled quite different exams. For both Math 221 and 222, Lecturer 1 gave two 
midterms and a final, worth 25%, 25%, and 40% of the total grade, respectively, with the remaining 
10% decided by the section instructor. These exams were given in the evening and the students were 
permitted to stay as long as they wanted to finish the exams. The final grades were determined 
according to a preset curve. 

By contrast, Lecturer 2 gave three midterms and a final for Math 221. The first and third midterms, 
worth 15% each of the total grade, were in-class exams and lasted for the 50-minute period. The 
second exam and the final, worth 25% and 30%, were scheduled for two-hour periods. For Math 
222, he gave 4 midterms, all in class and worth 15% each, and a final scheduled for a 2-hour peno 
worth 25% both semesters. The remaining 15% was decided by the section instructor. The exam 
grades were set according to a curve that was determined after the exam had been graded. 
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Our class observations (samples of lecture and section observations appear in Appendix B) and 
comments made by Instructors 1 and 2 indicate that Lecturers 1 and 2 also differed notably in their 
approach to the course. Lecturer l’s approach was strongly theoretical. He spent considerable time 
during lectures proving theorems and deriving formulas. He devoted relatively less time to solving 
problems that applied these theorems and formulas. His lectures, carefully organized and providing 
relatively few opportunities for questions from the audience, established a formal atmosphere. By 
contrast, Lecturer 2’s approach was to teach students how to understand calculus concepts by relying 
more heavily on problem-solving exercises. He sought to establish an informal atmosphere in the 
lecture hall, and succeeded in eliciting some student responses during a typical lecture. 

The students in both lecture classes generally arrived on time. The majority of the WES students sat 
together in clusters in the front rows, while most other students seemed to enter and sit down either 
by themselves or with one other student. During the lecture most students appeared to copy every- 
thing the lecturer put on the board, and pay little attention to their neighbors. Student attention began 
to wane noticeably during the last ten minutes of class, with most students packing up their books in 
the last couple minutes. When the bell rang, students in both lecture classes noisily pushed up their 
writing boards and left, even if the lecturer was still making a final point. 

Both lecturers held regular office hours throughout the semester. We have insufficient information to 
report how well they were attended. 

2. Discussion Sections and WES Workshops 
a. Discussion Sections 

DS students attended two 50-minute discussion sections weekly. While students generally stayed 
with the same lecturer for both 221 and 222, they rarely stayed with the same discussion section TA, 
and discussion group membership was random. The classrooms were undecorated, clean, quiet, and 
kept at a comfortable temperature. One side of each room had windows and two sides were covered 
with blackboards. A teacher’s desk and podium were placed at the front of the room, with some 30 
movable chairs with writing surfaces attached arrayed in rows facing the desk. Although WES 
researchers sat quietly in the back of the room, students noticed them and the instructor glanced at 
them frequently. 

In the discussion sections included in the study, the instructors typically asked for questions from the 
students on their homework problems. They then spent most of the 50-minute period answering 
those questions, and a small portion of the time explaining how the problem related to concepts 
presented in the lectures. While one student’s question was being answered, the other students 
watched and/or took notes, sometimes asking for clarification or volunteering other methods of 
solving the problem. Although the instructors often asked their students for input, such as how to do 
a certain step, the vast majority of the explanations came from the instructor. Each instructor was 
skilled at using the blackboard effectively, and presenting explanations. 

Other notable facets of the discussion sections were the quizzes, homeworks, pre-exam study ses- 
sions, and office hours. Both instructors gave in-class and take-home quizzes on a regular and 
frequent basis. The in-class quizzes were timed and usually given at the end of session. The instruc- 
tors collected and looked over the homework problems assigned by the lecturer, but each put more 
time into the preparation and grading of the quizzes than the homework. Usually several students 
attended every office hour, with some students coming on a regular basis. 
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While the planned instructional activities in the discussion sections in the study were very similar, the 
atmosphere in the sections varied as a function of the instructors’ personalities. For example, DS 
Instructor 1 used humor to make the time more enjoyable. Each session he cajoled the class to loudly 
intone the phrase, “Yeah calculus!” He constantly interspersed his explanations at the blackboard 
with humorous comments that often made fun of himself and the mistakes calculus students tend to 
make. The students obviously enjoyed DS Instructor l’s humor and antics. In contrast, DS Instructor 
2 worked with his class in a very well-organized fashion. He first listed on the board the problems 
the students wanted him to work. Then he systematically worked through the problems, occasionally 
adding explanations about what Lecturer 2 had meant. He maintained a very pleasant and responsive 
demeanor during his sessions. 

The students in discussion sections generally arrived on time, paid attention and copied the solutions 
that their instructor put on the board, and showed signs of waning attention during the last 10 to 15 
minutes. The classes ended promptly with the sound of the bell, and often three or four students 
clustered around the instructor at the end to ask additional questions. 

b. WES Workshops 

WES students attended three two-hour workshop sessions weekly. Workshop membership remained 
the same for both semesters except that 1 WES student who completed 221 chose not to take 222, 4 
were unable to take the 222 workshops due to scheduling conflicts, and 9 (6 men and 3 women) who 
were in regular discussion sections for 221 switched into WES. Both sections used the same room, 
with one scheduled during morning periods, and the other during afternoon periods. The room, 
located in a stately old building adjacent to the Math Building, was one of two large rooms located at 
the top of three flights of stairs. It was about twice the size of the DS classrooms, had windows on 
two sides, a blackboard and mural on the second side, a mural on the third, and posters illustrating 
great moments in mathematics history on the fourth wall. (The murals had been painted by WES 
students as one of the fall 1993 social activities.) The room had several built-in cabinets, counters, 
and bookcases, a noisy fan, and tended to be overheated. The furniture consisted of one long narrow 
wooden library table that could seat ten people, and six small conference tables that comfortably sat 
six people, a couple overstuffed chairs, an assortment of straight-back and rolling desk chairs, and 
several metal cabinets. The tables were placed at odd angles to each other, and the positions of the 
chairs changed as different students used the room. An assortment of papers lying around and empty 
soda cans left on tables added to the room’s informal and somewhat disheveled appearance. 

By the second week of the fall semester, WES Workshop 1 had 17 students, of whom 9 were women, 
and WES Workshop 2 had 18 students, of whom 7 were women. These numbers dropped to 15 and 
16 respectively by the end of the semester (3 of the 4 students who dropped were women). Of the 
original 35 students, 22 were members of underrepresented ethnic minority groups (including 
“targeted” Asians), and the 13 white students were from rural areas. As of the second week of the 
spring semester, Workshop 1 had 13 students, of whom 6 were women, and Workshop 2 had 22 
students of whom 7 were women. No students dropped from either of these sections. Nineteen of 
these students were from underrepresented ethnic minority groups. 

As the workshop period approached, a few students (fewer for Section 2, which met in the early 
morning) arrived early and sat around chatting or looking over their notes until the instructor, student 
assistant and rest of the students arrived. During the beginning of the two-hour period students asked 
about specific homework problems, and one of the instructors or another student answered the 
question, often going to the chalkboard to work out the problem. These large-group gatherings 
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usually lasted no more than twenty minutes and generally less in WES Workshop 1. Next the stu- 
dents would form into groups and work on the worksheets distributed by the instructor. 

The students were not organized into groups by any fixed method. The instructors sometimes as- 
signed groups. For the most part, and particularly during the second half of the fall semester and all 
of spring semester, the students formed their own groups. These self-formed groups remained fairly 
constant, although some people occasionally switched from one small group to another. 

The remainder of the workshop time was spent on worksheets. While a few students would work on 
problems alone and only ask their peers or instructors for help when they got stuck, most worked by 
interacting on a continuous basis in teams. They talked about different methods of solving problems 
and compared answers. If one person solved a problem, she or he would explain it to another person 
who did not understand. Occasionally a student from one group wandered off to see what another 
group was getting, and then reported back to his or her own group. The WES researchers, sitting off 
to one side taking notes or wandering from table to table, were largely ignored. 

Once the workshops broke into small groups, the ambiance was lively and very informal. Joking 
comments, laughter, and exclamations of interest were heard frequently, although mostly students 
remained focused on the papers in front of them. Listening to conversation during our classroom 
observations, we heard small group interactions shift fluidly from calculus to discussions of social 
matters such as football games, parties, and personal concerns. Some students either brought or left 
the room briefly to get sodas. On a few occasions, such as after exams, WES Instructor 1 brought 
snacks which students wandered over to get at intervals. Between their work and social interactions 
and the constant activity in the room, students barely noticed visitors such as the researchers. As the 
period came to an end, different individuals and groups packed up and left over a ten-minute period, 
with a few students staying after the instructor left to continue talking or working. 

WES Instructor 2 gave no quizzes during Math 221 but gave some during Math 222. WES Instructor 
1 gave one practice exam and a couple quizzes in the fall, and a couple quizzes in the spring. Both 
instructors used these quizzes not to help determine section grades, but to help students determine 
where they were weak and to give them practice at taking tests. WES Instructor 1 did not participate 
in grading exams, but did give a discussion section grade worth 10% of the total grade. WES Instruc- 
tor 2 helped grade the exams, but did not participate in setting the curve. 

Like the other students in their large lecture class, the WES students were required to complete the 
homework problems assigned from the textbook. (The students did not do homework problems 
during their workshop periods.) Following the lecturer’s schedule, students gave their homework to 
the instructor, who in turn gave it to a grader and returned to the students once processed by the 
grader. 

As did the DS Instructors, the WES Instructors and student assistants (SAs) held official office hours 
and also were generally available to meet students at their convenience. Three or four students 
visited WES Instructors 1 and 2 each week, and a slightly larger number visited the SAs. 

Various kinds of informal WES student social interactions had been planned as part of the WES 
Program. Students who enrolled in WES during summer orientation were offered the option of living 
together on the same floor of a dormitory, and 14 chose this option. The program director had 
anticipated that this floor would foster evening study groups and have occasional planned events. As 
it turned out, only one planned event transpired: a speaker came to the WES dorm floor one evening 
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at the beginning of the fall semester to discuss mathematics in industry. We do not know if WES 
study groups formed on the floor during the evenings. 

Another kind of WES social activity consisted of planned events for the entire WES group. During 
registration week a pizza party was held for the WES students, students who were invited to join a 
WES Calculus 221 section but who placed into Calculus 222, plus members of the Math Department 
and various College of Letters and Sciences and central administrators. In September there were 
several informal gatherings to paint the WES room. Instructors 1 and 2 arranged a trip for their 
classes to visit an exhibit of Escher’s work in Madison and a bus trip down to Chicago to visit the 
Shedd Aquarium and the Museum of Science and Industry. A final WES party was held at the end of 
the spring semester. 

A third type of WES social activity was the pre-exam review sessions held by the S As. These took 
place in the WES classroom on the two nights preceding each exam. The students usually worked on 
problems in groups and the senior peer helped in the same manner as she or he did in the workshop. 

It was not uncommon for the students to order pizza while they worked on the problems. Approxi- 
mately two-thirds of the students attended each review session. 
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B. Learning Processes for Students 

This two-part section features the learning processes that characterized the WES workshops and 
discussion sections. The first part presents characteristics of the learning processes as described by the 
students who were interviewed. The second part of this section features the researchers’ analytic 
generalizations. 

In each part we present findings from the DS students first, and then turn to findings from the WES 
workshops. The quotations appearing below represent perspectives found across interviews, and were 
selected because they articulate these themes with clarity. We note when quoted excerpts are from 
interviews with the students who switched from a discussion section into WES for Math 222, and 
from WES 221 students who had to switch out of WES for Math 222 due to scheduling conflicts. 
These “switchers” drew sharp contrasts and consistently and strongly preferred the WES approach. 
Their views strongly confirm the experiences of the students who attended WES workshops both 
semesters. 



1, From the Students’ Points of View: Key Features of the Learning Process 
a. Discussion Sections: Individualism 

Analysis of the DS students’ interview transcriptions reveals three themes organized under the rubric 
“individualism.” These themes are that social dynamics are a function of instructor-student interac- 
tion, competition is a source of self-esteem or self-blame, and learning entails a solitary student-to- 
“content” relationship. 

Social Dynamics Are a Function of Instructor-student Interaction 

We learned from DS students that the nature of social interactions in their sections are, for the most 
part, a function of the interaction between the instructor and the students. Characteristic of the 
responses we received upon asking about the nature of interaction among students in the discussion 
section was this remark: “It’s not where you are going to turn to your neighbor and ask him to answer 
a question because it just seems like you keep your attention on [DS Instructor 1] all the time...” 
Another student put it this way: 

S: It’s just the way [DS Instructor 1] teaches, I mean, people are riveted to him. He is, he really 

keeps your attention because you don’t know what he’s going to do next, I mean, maybe 
he’ll do something funny, he’s just kind of unpredictable up there, but you know, the class is 
really riveted to him and so when he wants to answer questions he can do it real easily. 

I: I see, because he doesn’t have anybody drifting off. 

S: Right, he has control. 



To be fair, we note that DS Instructor 1 had an extremely strong reputation among the students and 
that his section’s average GPA was among the highest of all 66 Calculus 221 discussion sections. 
Many students indicated that DS Instructor 1 was remarkable for his ability to keep their attention and 
communicate concepts. They said they felt lucky to be in his section, contrasting him with many 
other TAs whom they felt clearly placed students in disadvantageous positions within large lecture 
classes. 



Students in the discussion sections generally viewed their discussion sections as experiences where 
they listened to the DS instructor and then tried on their own to learn the material outside of class. As 
one student noted, “...most of what we learn seems to be what we teach ourselves out of the book or 
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what our TA teaches us. The TA sort of wraps up what the professor says and then you’re kind of on 
your own.” 

Students generally took for granted that the character of the social interactions in the discussion 
sections consisted of interactions occurring almost exclusively between the instructor and each 
student. When asked to reflect on the character of their discussion section’s social dynamics, most 
students seemed p uzz led and indicated that it was normal that each student just focused on the instruc- 
tor, quietly waiting his or her turn to have a problem worked out at the board. 

A corollary of these students’ implicit view of the class as a set of individuals whose primary con- 
nection to one another is a shared instructor is their view that group work would waste their time. 

Most students responded to our questions about group work with negative reactions. The following 
discussion is representative of this dominant view: 

I: ...what do you think about working in groups on hard problems... 

SI: I don’t like working in groups. 

I: How come? 

SI: To me I feel that I can figure most stuff out for myself, given time. And it seems like other 
people can’t. You know it’s just the way I am. I am impatient. 

S2: That is the same way I feel too. Most of the experience I have with working with groups — 
we did that a lot in high school, it seemed like I was always the one who was doing the ex- 
plaining. 

S 1 : Plus down here I don’t really want to help other people. 

I: Why is that? 

SI: It is a competition thing. I just want to help myself, not anybody else. 

S3: See, if I was in a large group, it would either be, like, I’d feel I’d be doing most of the work 
or else I wouldn’t be doing nothing. I would just be sitting there letting everybody else do it, 
and I wouldn’t get anything out of it. I like one-to-one or by myself. 

Many students felt that they needed to work on their own, with guidance from the instructor, to 
develop an understanding of the course material. As they saw it, the discussion sections were struc- 
tured to meet this need. 

A few students, however, expressed dissatisfaction, comparing their DS sessions with the much more 
interactive classes they had had in high school. For example, when asked what she thought about her 
discussion section in general, one student said, 

It’s really quiet. Nobody talks. I don’t even know a lot of the people in my class. People 
don’t ask questions very much. I feel like I ask questions a lot. So I don’t really know that 
many people in my class, which I wish I did so that I could work together with them. 

When asked about whether a different ratio of males and females changes the dynamics of the class 
this same student further characterized the class dynamics by responding, “We have no dynamics. 
Like [DS instructor 2] does a lot of talking and then he’s like, ‘Any questions?’ I don’t know if 
people are hesitant to ask or what it is.” When asked about her expectations of the discussion sections 
she showed concern about the lack of student-to-student interactions: 
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I expected [the discussion sections] to be small, but I didn’t expect them to be so 
repetitive of lecture. I thought it would be more discussion. I thought it would be more stu- 
dents talking, students asking questions, students clarifying points by asking questions. And 
a lot of times our discussion seems like a minilecture. And that I didn’t expect. 

A similar concern was offered by a student when asked about the basic atmosphere of the discussion. 

It is kind of a reserved atmosphere.... it isn’t the kind of class where you get to know peo- 
ple’s names and you work together on things. In my previous calculus class [in a community 
college setting]... we were put into groups of three and then assigned all the problems at the 
end of the chapter... And at least that was a way that we could work with other people and 
get to know what they were doing and find out how they approached problems. 

Competition Is a Source of Self-Esteem or Self-Blame 

The role of competition raised in an interview excerpt above is also a theme running through DS 
interviews. To be sure, this theme was not salient for a few of the students in our sample. These 
individuals perceived their discussion section not as a competitive but as a neutral environment in 
which each student was just trying to do better within her or himself. A more widely-shared view, 
however, was that the atmosphere was competitive. Students who saw it this way and perceived their 
performance as below that of others were inclined to blame themselves for either inadequate study 
habits or lack of “math ability.” By contrast, those students who knew they performed better than 
their peers would note that this knowledge gave them confidence and made them feel good. 

In describing these dynamics, some students conveyed that they felt the atmosphere was intimidating. 
For example, in response to a request to reflect on the social dynamics of his discussion section, one 
student said, “I’m worried that someone will ask a question I don’t know the answer to,” indicating 
that he was judging his own competence or ability to make the grade by comparing himself to others, 
and expressing a fear that he would find himself inadequate. A further example of this competitive 
perspective was illustrated when a student stated, “In discussion section when somebody will jump up 
and yell the answer and you’re like, ‘Where did he get that?’ I just feel kind of like being left be- 
hind.... I kind of withdraw and work by myself so I can work things out.” For this student and his 
feelings were echoed by many others — uncomfortable feelings arising from competition were par- 
tially resolved by retreating into more solitary work. 

Learning Entails a Solitary Student-to- i( Content” Relationship 

Closely related to the theme that competition is a source of self-esteem or self-blame is the theme that 
the course is a kind of test during which each student discovers whether he or she is one of the 
students who is succeeding in the course (earns As or Bs), or is one of those who doesn t get this 
stuff.” While opinions of the lectures varied— largely as a function of each individual’s capacity to 
“keep up” — most students conveyed the idea that the explanations and examples were presented 
clearly and thus if a person studied hard, whether he or she “got it” was a sign of whether he or she 
has talent for math. This discovery was experienced as a solitary process, as one studied and assessed 
one’s understanding, and then received exam grades which (assuming the exam was perceived as fair) 
established one’s standing in the class. Those who studied hard, felt they were “getting it,” and then 
received good grades, expressed great relief and satisfaction that they were in control of their calculus 
experience. Typical of remarks made by strong students is the following: 
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S: I did really good on both of the exams, and I expected to get a really bad second exam, be- 

cause, to be honest, I attended 2 lectures between the first and second exam. So I missed like 
ten or so, which is not really good. So all that I did was study from the book, and I still got a 
really good grade and that kind of amazed me. 

I: Was going to [DS Instructor l’s] section effective? 

S: Yah, that’s for sure — all the problems that we went through with him. 

I: So you still did the homework? 

S: Yah, I did the homeworks. Not every time but usually. And, attending [DS Instructor l’s] 

section was a good thing to do... 

I: And do you have any particular plans for how you are going to study for the final? 

S: I don’t know. It’s my basic standard stuff that I am doing. I am going to take the book and I 

am going to review all of the chapters that we went through, starting from 1.1 all the way up 
to 7 or 6-something. And that’s basically it. I mean I am not go in all the detailed stuff 
again. What I am going to do is try to do the exercise sets, and if I feel that I am lacking 
something than I go back into the actual chapter, and read the stuff again and try to work it 
out. 

I: And you will also be working by yourself? 

S: Yah, I don’t like working with people in math. Sometimes I do. Sometimes I sit together 

from people in my dorm, you know, there is a guy in 222, and another in 223, and since I 
had math in high school, pretty advanced math, I can sometimes help somebody in 223, a 
real minor problem, and they can help me. But if I am really studying, then I need to be by 
myself. 

It is of note that strong DS students experienced their learning process as a solitary and affirming 
(and, in the case of the student quoted, subtly competitive) student-to-content relationship. By 
contrast, students who were not “getting it” even though they studied hard 2 expressed great frustration 
with their struggle to grasp a body of material which eluded them. These students also experienced 
their learning as a solitary student-to-content relationship, but in contrast to the strong students, their 
experience was alienating instead of affirming. This idea is communicated in the following excerpt: 

S: They have so many pages before the pages that explain what it is about and then you go to 

do the problems and it’s like, there is no example to show how to do it, and you have all 
these problems and you don’t know how to do it and I’m just sitting there trying to figure out 
all these things and it never works and that’s pretty much how you get frustrated. Because 
we can read something and then just look at it and figure it out. But if I don’t have any ex- 
ample, if I have to just sit there and look, it takes me a long time. But once I can see some- 
thing, I can apply it. ... 

I: If the course could have been taught differently, like if you were going to design how it was 

taught, how would you have designed it? 

S: I don’t know. That’s hard to say. I guess I’d try to smooth things out. I mean, for me, the 

lectures and the discussions were always completely different. They never really would ever 
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try to [synchronize]. I don’t know if it was the T.A. doing what he thought we should know, 
but as far as I’m concerned, I think he should try to explain what was done in lecture. It was 
like, “You’re on your own.” You just take notes and did what you [could to get something] 
out of the lecture. And then in our discussion he would do something else. 

I: Okay. What kind of things went on in lecture that were different from the discussion sec- 

tion? 

S: It was just, um, our professor would just do a couple problems on the board, and he would 

go through them real fast, and he’d be like, “Try to keep up. If you can’t you are out of 
luck.” And that is pretty much how it went. He would just get up there and do a couple 
problems. Seriously, we would do just two problems the entire time we were there. 

b. WES Workshops: Group Work 

Upon analyzing the WES students’ interviews, we articulated a set of themes organized under the 
umbrella of “group work.” These themes are that: work groups are a source of support; WES fosters 
a sense of safety; group work generates a sense of belonging and being special; group work facilitates 
learning; instructors and SAs are there for you when and only when you need them; challenging 
worksheets facilitate learning; and work groups vary in character and effectiveness. 

Work Groups Are a Source of Support 

We learned from the WES students that an essential characteristic of WES is the work group. Most 
WES students stressed that a key reason group work was important was for the support it provided. 
One student described her experience in her work group in the following manner: 

It’s great. We’re a family, we’re all in the same boat. That’s probably what I like the most 
about it is the people. I love the group. 

Another student expressed the outcome of group work for her in the following manner: 

And I felt like if I hadn’t been in this group I would have done much worse and I wouldn’t 
have understood, because — I’ve been talking to other people in the same [lecture] class and 
they’re just like, “My T.A. is terrible” or whatever, and it just seems to be luck whether or 
not you understand. And here I just feel, like, really solid and I have a support group. With 
a lot of people that I talk to, well, I don’t know that many people in the [large lecture], but 
from some of the people I’ve talked to they’ve said that there were problems on the test that 
they “had never seen before and this was crazy,” and for me these were the sort of problems 
that I expected to be on the test. 

Responding to the question, “What are your impressions on how the WES Program is going?” 
another student said, 

I don’t know.... I felt really glad I was in it after the exam ... especially since we had, I 
thought it was something extra that we had the two study nights with our, well not T.A., but 
one of the people who helps us, like a T.A. [senior peer] had met with us and helped us work 
the problems and I just felt like I was getting extra support and that I had, y’know, all these 
people to work with who were, [who I] could just meet with. 

This atmosphere of support includes playful competition, particularly among the men. For example, 
a student described his group’s activity in the following manner, “For the most part we walk into our 
discussion, she basically hands out worksheets, we get in a group of three or four people, and we just 
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work problems. And we argue with each other and say, ‘This is the right way,’ ‘No, this is the right 
way.’” 

WES Fosters a Sense of Safety 

It is important to note that within the first few weeks of the semester particular groupings of students 
cohered into self-identified work groups, each, with its own attributes. As the students explained it, 
this sense of group cohesiveness was associated with the development of an ethic of mutual support 
and of being accepted as a valued contributor capable of thinking. Students explained that the 
process of cohering as a group helped them to feel comfortable about asking questions and expressing 
their need for assistance from other group members as well as from the WES instructor or SA. In the 
following excerpt a student from WES Workshop 1 presents this characteristic of her WES workshop 
by contrasting it with discussion sections: 

SI: I feel like if I’m having trouble I can call on any of the other students and say, you know, 

I’m having some problems, would you mind studying with me, and I feel that if I ask that, 
anybody would say sure! 

S2: Definitely! 

I: And in other big classes you don’t feel that way... 

SI: Well, no, I mean definitely if I was just flopped down in a discussion [section], I’d be like, 
“Who’s the person sitting next to me?” you know. I mean [you’d be thinking], “How much 
more do they know? Are they going to be like, ‘God, you’re dumb to not get that.’” 

A similar effect was described by a student in WES Workshop 2: 

Yesterday we were working through these inequalities with absolute values. That was just 
horrible. I was really, really lost and I got really frustrated and I was like, “I’m in a calculus 
class. I hate math.” We were working together and I was so upset. I was like, “I can’t fig- 
ure this out.” I felt so dumb. What I like about this class is I was comfortable enough that I 
could express the fact that I was having trouble.... And one of these instances Senior Peer A, 
who was working with us that day, he came over and he sat down with me and was like, 
“OK, let’s do the problem individually.” And I think that was really good because I was 
comfortable enough to do that. 

I don’t think in a regular discussion, like any of my other discussion classes, that I would 
have felt comfortable saying, “Hey, I’m struggling and I need help.” I think that because we 
did know each other beforehand, with the party and all, we were all working together. This 
is probably the only class that I would probably feel comfortable doing that. 

The WES students confirmed the DS students’ perspectives that the discussion sections are intimidat- 
ing by explaining that in DS environments it feels risky to admit to needing help. For example, one 
student highlighted the differences between WES workshops and discussion groups as follows: “I 
think that you feel more comfortable to experiment and to think and to try new methods and that sort 
of confidence and that sort of comfort level is one of the differences for learning, I think.” 

In sum, it was the contrast between WES and their other experiences that led WES students to notice 
the more competitive and individualistic atmosphere of their other classes. In articulating this con- 
trast, the WES students focused on the importance of having a supportive environment where they 
felt safe to admit there were areas in which they needed assistance. 
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Group Work Generates a Sense of Belonging and Being Special 

In addition to the sense of safety derived from group work, WES students said that their groups gave 
them a feeling of belonging which mitigated their sense of being “anonymous” in the large calculus 
lecture as well as at this . huge university. This sense of belonging is illustrated by the following 
comments from a focus group interview. These students were presenting their reactions to their first 
examination. 

S2: ...I did really well on it and I was looking at why I did well and I just think it has a lot to do 
with being in the group, because I just feel like there are so many people in that class, I 
would feel like another number there and then I just feel like we have a little group, not that 
we’re special or anything but just that it means... 

I: You are supportive of each other. 

S2: Yeah, it’s just a little group within the whole big lecture. Otherwise I’d feel almost like I 
was competing with everyone else in there and somehow, there’s so many people, and I was 
just competing and now I don’t feel like that... 

S 1 : And when the tests were returned, I not only wanted to do good myself but I had a feeling 
that, just the whole group — you just want us all to do good. 

I: Who wanted you all to do good? 

S2: I was hoping we would all do good ‘cause it was like, ‘here come back our tests — let’s hope 
we all showed good and we all did our best.’ So I actually don’t feel like I’m in a calculus 
lecture with, how many people. I feel like I’m in the WES class and that’s all there is. 

Group Work Facilitates Learning 

The material quoted above su gg ests that WES students moved seamlessly from describing the impor- 
tance of their work groups as a source of support to describing their work group’s central role in their 
learning of calculus. This central role was made evident when student after student explained how 
each member of his or her group usually had something different to contribute to the problem solving 
process. For example, one student discussed how combining the different strengths of the group 
members facilitates the learning process: 

You notice that some people will be able to solve one problem and another person won’t be 
able to solve it. Like I’ll solve three problems in a row, I’ll get it, and then the next one... it 
might be something easy, but it will just be something that I forgot or a way that I... and then 
someone else will say something that will just snap... 

I think that groups are really important because, say, he’s really good at trigonometry and 
maybe I’m really good at basic algebraic skills or something. And maybe someone else is 
just really good at remembering stuff. So when you put all those skills together, we work it 
together, without leaning on a book or without leaning on a professor, we get it from each 
other. And it just makes everybody better... You see yourself improving in areas that you 
wouldn’t think... I think you learn best from other people. When you see someone at your 
own level, then it actually seems accomplishable. 

Another student, when asked to describe what the WES groups are like, responded, “Sometimes I 
really get into it because you get different ideas and maybe different ways of doing things that maybe 
you’re not used to or you don’t know....” A third described how his approach to doing math has 
changed in the following terms: 
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I think that it happened by, I started doing a problem one way. [One person] sits there and 
she starts doing the problem another way. Then [another person] does it a different way. And 
we all start different ways and we’re like, “Oh, I didn’t think about doing it like that. What a 
good idea.” Or like that, and then the more ways that you see different ways of doing a 
problem, then you sort of pop them in your memory bank, then the next time that you see 
that problem with different numbers and different variables, then you might use that method. 

These passages illustrate the essentially cooperative framework of the WES work groups. The 
students explain that a mutual dependency develops in these groups through each group member 
contributing his or her differing skills and knowledge. These excerpts also suggest the idea that some 
students feel that they learn better from peers because they are on the same level. 

Instructors and SAs Are There When and Only When Needed 

We asked only one question specifically about instructors (“Does your instructor’s personality 
influence your experience in this course?”) and one about the student assistants ( What do you think 
about the undergraduate teaching interns?”). We anticipated hearing students reactions to their 
instructors when they answered less pointed questions, such as what was most and least helpful about 
the course, what two or three times from the course stand out, and what they will remember about the 
course a year hence. 

While the DS students invariably offered evaluative remarks about their DS instructors whether or 
not we were asking for opinions of them — the WES students only described their instructor when 
asked pointedly. For example, the following interchange during a one-on-one interview began with a 
question about the WES student’s impression of her instructor and the senior peer. 

S: I like them all, I really do. 

I: Do you think, I mean, what is it that works about them, why do you...? 

S: Because they don’t ever tell you what to do, they always, they never tell you an answer, it’s 
impossible. 

I: Does that frustrate you? 

S: Yeah, but that’s good, because you don’t really leam if they just tell me the answer. 

I: Okay, anything else about them that sort of sticks in your mind? 

S; Their intelligence. You can tell that, 1 mean, it’s not just intelligence, it s not just academic 
intelligence, it’s that they like what they’re doing and so they can get into it and it’s not just 
math problems, and they make it interesting to you. At least, now take [Senior Peer A], you 
can tell he really likes what he’s doing. And that’s why it’s easy for him to understand and 
that’s why it’s easy for him to explain. 

I: How about [WES Instructor 1]? 

S: She’s really nice. 

In response to pointed questions like the one that initiated the excerpt quoted above we learned that 
the students genuinely liked their instructors. Some students observed that they felt their instructors 
really cared about them and had high expectations of them. In addition, some students in Instructor 
1 ’s section commented that they enjoyed having a female role model. But we had to press students 
for their opinions. It was as if the instructors just didn’t come to mind much. 
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Significantly, unelicited comments about the WES instructors only arose in the context of describing 
how their groups worked. Typical of such remarks is the following: 

...if [our small group] can’t agree the [instructor] will come over and he’ll ask, “Well, what 
do you think about this?” They’re really good about not telling you too much about any- 
thing. They like to let you figure stuff out and think for yourself. 

In general, the WES students gave the impression that the instructors were excellent, mostly because 
they were there when — and only when — they needed them. 

The students also conveyed the idea that the SAs were also “there for you when and only when you 
need them,” but made many more unelicited comments about these individuals than about the older 
instructors. In these comments they conveyed that they felt more comfortable with the SAs, more 
able to identify with them, and often more able to learn from them. For example, the members of one 
focus group explained that the first instructor they preferred to turn to for help was the SA. In 
response to the question of why, they said: 

SI: I just think that he seems really excited about math to me, and he is definitely very con- 
cerned about each one of us understanding this and being as excited about it as he is. And I 
just think he is great. He is really exciting.... 

S2: Not that [Instructor 2] isn’t. I think that maybe it’s just that he’s closer to our age. 

S3: Yah, I was just going to say [that]..., and he’s just right there. 

S2: It’s maybe more of an equal vs. mentor. 

A student in a one-on-one interview communicated his appreciation of both the instructor and the 
SAs, with a special twist on the importance of having taken calculus within the last couple years. 

S: ...the good thing about undergraduate TAs [SAs] is that they understand our problems and 

stuff. 

I: You mean, they understand where you might get bogged down? 

S: Yeah. 

I: Okay, so let’s say somebody was going to be sick, would you rather it was [the Instructor] or 

the [SA]? Would you rather the [Instructor] be the one that didn’t come? 

S: Either one, it doesn’t matter. Because they have their different ways in approaching things, 

and I don’t think you can really compare the two. 

I: Okay, but you like both, you like having both around? 

S: Yeah. 

I: Would you say it makes a good balance? 

S: I like being balanced rather than just one extreme and the other extreme or an either/or situa- 

tion. 

I: So [the Instructor’s] sort of perspective ... is actually kind of a good thing to have as well? 

S: Yeah. Another thing, the undergraduate assistants have just gone through the same type of 

stuff that we’re going through recently, relatively at least. Instead of maybe 40 years or 
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something, I mean, I don’t think she’s [the Instructor] been doing this undergrad stuff for 
quite a long time. 

I: Right, like 25 years or something, maybe she’s forgotten. 

S: Maybe she’s forgotten experiences, but she’s dealing with undergraduates in terms of just 

teaching them, but not quite as recent as the undergraduate assistants. 

Challenging Worksheets Facilitate Learning 

Much like the WES students tended to take their “there for you when and only when you need them” 
instructors for granted, they also took their worksheets for granted. They were very aware that the 
worksheet problems were more difficult than the homework problems, or even most of the exam 
problems. But they rarely commented on the worksheets unless we actively elicited their views. 

Once we got students to reflect on the role of worksheets, they provided the kind of analysis that 
appears in the following dialogue. 

I: What about the worksheet questions? Give me an assessment of them. 

S: I think they’re very hard but I don’t know if that’s good or bad, I mean, I guess from [the 

instructor’s] point of view if you can do the hard problems you can do the easier ones... You 
can’t usually finish the whole worksheet. I mean today we did, but that was like, that was 
odd. We usually finish like half the sheet. 

I: You mean, like turn them in at the end of the.... 

S: No, you don’t turn them in... I think the worksheet does pretty good, but I mean I wouldn’t 

be able to do it by myself, that’s the thing about it. 

I: So tell me, what if you had some problems and you’re asking some other people about how 

to do it and you feel like you couldn’t do it yourself. When you leave, do you feel like, “If I 
had to do that tomorrow, I could do it myself’? 

S: Yeah, you feel like you have a better understanding of the [theory]. I think that’s why we 

have those kind of problems so you can understand why instead of just, you know, doing 
problems, doing a whole bunch of problems, you have 4 or 5 hard ones to make you under- 
stand why these problems are done the way they are so you can do them the next time by 
yourself. 

I: So it takes you into a new sort of ... 

S: Right, it gives you, yeah, I understand the theory behind it. So in that way that’s good. 

This student’s analysis highlights how the hard problems force students to pool their efforts to 
develop deeper understandings. Another student communicated a similar appreciation of the value of 
hard problems, and also expressed an understated pride in their difficulty, explaining that he found it 
“a little frustrating” that the exams didn’t test them in ways that let them show everything they 
learned doing the worksheets. 

I: What do you think about the worksheets? 

S: Worksheets? I think some problems are probably harder than the ones in class.... Well, not 

all of them are harder. 

I: Now do you find that useful or do you find that irritating? 
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S: A little of both, because it’s useful to know that you can do things that are higher than your 

expectations, it’s a little frustrating when you... can solve the hard [problems] on the work- 
sheets... but you can’t solve the easy problems on the test. 

I: Oh, does that happen? 

S: Sometimes. 

I: Now explain how that would work. 

S: Sometimes you think of things that are a little beyond what’s needed for the easy problems, 

like maybe something that’s too obvious and you overlook that... because you’re more in- 
volved with the more difficult things. 

The WES instructors affirmed that, from their perspectives, the worksheets were critical to the 
effectiveness of the WES approach. 

Work Groups Vary in Character and Effectiveness 

We also learned that, while the work groups provided support and a sense of safety and belonging for 
their members, some groups were more effective as learning communities than others. One important 
difference among groups was that some functioned as loose coalitions of people who mostly worked 
on their own and checked with each other from time to time, while others constantly worked together. 
Another difference was that some groups facilitated the development of very high expectations and a 
shared attitude of commitment to really master calculus, whereas this effect was less noticeable for 
other groups. Certain groups, for example, made it part of their identity that they were a group that 
“doesn’t give up,” no matter how hard the problems are. In addition, some of the more reflective 
interviewees noted that the all-women group — whose members more openly enjoyed the support they 
gave each other — were different from the all-men groups, whose members expressed their apprecia- 
tion for each other through a kind of “tough” banter and playful competition. In any case, each group 
had its own character, which was a product of the synergy of the personalities of its members. 

In summary, the WES students communicated that the support of their group helped them to feel safe 
about expressing their need for assistance, and gave them the confidence to utilize interdependent 
group dynamics. Drawing on their new confidence and utilizing one another’s different strengths, 
they grappled with difficult problems and tried new ways of problem solving, which gave them not 
only more confidence but a sense of pride and belonging. 

2. From the Researcher’s Point of View: Analytic Generalizations 

We have structured our analytic generalizations about the learning processes in both discussion 
sections and WES workshops under the following three headings: sources of affirmation, approach to 
problem-solving, and attitude toward learning. 

a. Sources of Affirmation 

Discussion Sections: “Getting” the Instructor’s Solutions through Observation 

In the traditional discussion sections the process of learning mathematical concepts is controlled by 
and filtered through the instructor. The instructor leads the students through the material from lecture 
and works through problems based on student questions. The students interact with the material 
largely as observers of the instructor’s process of problem solving. For example, in a videotape of 
discussion Section 1 the instructor attempts to show students that there are multiple ways of solving 
problems by describing and then illustrating different ways of approaching a problem. In the process, 
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he attempts to have students contribute ways of solving the problems. He responds to what they say 
by filtering — and judging — their contributions. The instructor, concerned about not leading the 
whole class on a tangent or confusing most of the students while he attends closely to one student’s 
particular insights or misunderstandings, is thus forced to rather quickly reinterpret any one student’s 
contribution so that it is — as far as the instructor sees it — right. 

This process, repeated throughout the semester, leaves the students dependent on the instructor for 
affirmation of mathematical solutions. A common student response to this instructional environment 
is to view knowledge as something that comes from a more authoritative other person, not as some- 
thing they develop for themselves. Thus, each student’s source of affirmation is narrow, coming 
almost entirely from the instructor. Contributions from students tend to be “normalized” by the 
classroom dynamics, thus vitiating student ownership of their learning. The structure of the learning 
environment tends not to encourage the development of individual student insights, and is not de- 
signed to respond to students who have different learning styles. 

This point is illustrated in the following quote from a student in discussion Section 1. (As noted 
earlier, DS Instructor 1 had an unusually strong reputation among the students.) 

S: It is his, um, the way he encourages me to look at the problems, to, not to lose my cool when 

I am solving problems. He always makes things so simple. He is just standing there and you 
think that it is the hardest problem and then he tries to approach this problem with your 
thoughts, and then, he gives you the answer. I talked to [another interviewer] about this: 
there is no other way, because we have 50 minutes twice a week. So there is no way that we 
can have, like, people go to the board and have them explain what they did and where they 
are stuck or something. But the way that he approaches the problems, and of course his char- 
acter encourages too. 

I: When you say that he approaches it from your thinking... ? 

S: Well it is like, he tries, not to manipulate, but to come in on the right path. It is like you look 

at a problem and you can’t solve it, and he starts with the really basic stuff of this problem 
and then if you even can’t start with this stuff then you are lost on the problem, but he helps 
you get on the right track, you go “oh, ok.” 

WES Workshops: Generating One’s Own Solutions by Interacting with Peers and More- 
Experienced Others 

For students in the WES workshop the process of doing math is predominantly peer-oriented. 
Actively engaging in problem solving processes with their peers, the students find themselves offer- 
ing suggestions and affirming and correcting one another. Through this process they develop confi- 
dence that they understand concepts and can solve calculus problems both as a group and as indi- 
viduals. This process was described in the following manner by a WES student: 

S: I usually work in the same group with these kids and we usually get a lot of work done. It’s 

a good atmosphere to work in. I talk to most of my friends [who are] in [a traditional] dis- 
cussion and they just sit there quiet and the teacher just does problems on the board all hour. 
They memorize and copy, that’s all they do. In this class you’re actually learning something 

I: Is that because of the group work or because of the problems? 

S: I think because we get worksheets every day. I think that’s what it is. Because when those 

[DS] kids go to discussion, the teacher just does the problems that the professor does in the 
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class — -just does them over again, or tries to explain them more in depth. They just end up 
memorizing and copying things down. We get in groups and we discuss the problems and 
do them together instead of just memorization and stuff. And there’s no answer key. So you 
keep going until you’re sure you have the right answer. When we find the right answer, 
you’ll know you find it because there’s three of us in the group, and two of the three should 
have the correct answer. And if the other one is still unsure we go over it again. If you keep 
applying the laws and principles of calculus, you’ll come up with the right answer. 

This process contrasts significantly with the learning process in the discussion sections. For help and 
affirmation DS students depend almost exclusively either on themselves (cf. Solitary Student-to- 
Content Relationship, p.23) or their instructor. Some DS students acknowledge that they occasion- 
ally ask more advanced students in the dorm for help. A small proportion take the initiative and seek 
“extra help” during office hours or in math lab. However, in casting these one-on-one interactions as 
“extra help” they convey their sense that the normal learning process should not include this kind of 
work with others. By contrast, students in the WES environment primarily depend on one another, 
both to generate understandings and solutions and to get affirmation for having good ideas and 
producing good answers. 

A corollary of the peer-oriented WES learning process is that the instructors’ role is experienced as 
secondary. We hasten to emphasize that the instructors play a critical role in WES. It is they who 
provide an environment in which student-generated learning can thrive, an environment in which 
students are able to place instructors in the background. The instructors and the SAs actively work 
on creating this environment. They self-consciously take the role of facilitative “educator” ( educere , 
Latin for “to lead out or draw out”) rather than the traditional role of “instructor” ( instruere , Latin for 
“to build in”) (Wilshire 1990, pp. 21-22). Rather than soliciting student questions and providing 
direct answers, they wait until students seek their help, draw students into consideration of new 
perspectives, and then leave. As quoted above, a WES student discussed the process of group work 
and the role of the instructor in the following manner: 

...if we can’t agree the [instructor] will come over and he’ll ask, “Well, what do you think 
about this?” They’re really good about not telling you too much about anything. They like 
to let you figure stuff out and think for yourself. 

Such practices allow the students to continue, in their groups or on their own, to generate their own 
answers. Therefore, though WES students frequently turn in frustration to get help from their senior 
peer and faculty member, and greatly value their knowledge, these more experienced individuals 
remain in the background. 3 

In the WES learning environment, a student’s resources for learning, sources of affirmation, and 
hence sources of authority are vested in many people: her small work group, her other WES section 
peers, the SAs and a professional mathematician. These various student-student, student-SA, and 
student-mathematician interactions offer different types of affirmation to the student. Our classroom 
observations, the videos and interviews with both students and WES instructors confirm that, taken 
together, these various relationships draw students into the doing of calculus. Each student par- 
ticipates in the workshop at complex experiential levels, with more or less greater involvement 
depending on his or her personality and needs. Working within an environment that is lively, they 



3 Note that as of 1994-95, the instructor role will be occupied by graduate students. 
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continuously make connections— both mathematical and social. The students and instructors together 
“own” this instructional space in which the students actively learn mathematics and simultaneously 
provide one another new identities as capable mathematical thinkers. 

Metaphorically, all students work along paths that climb in elevation to the heights of understanding 
calculus. The discussion sections move up a path on top of a ridge line. By contrast, the WES 
students move up a path in a ravine. The forces students experience when they diverge (or get 
“displaced”) from these paths are very different. In the discussion section, the resulting forces act to 
move students further from the ridge line; there is nothing to catch them if they stumble and fall. In 
the WES section, the resulting forces are restorative. Most DS students, therefore, have two strug- 
gles — one to stay on the path, and the other to move to the heights of calculus. 

b. Approach to Problem-solving 

Discussion Sections: Understand the Mathematical Formulations by Listening to Clear Lectures 
and Working Hard (by Oneself) 

Our analysis of the DS approach to problem-solving is based on an excerpt from an interview with 
Lecturer 2, who has many years experience teaching the calculus course. 

...what really counts is getting students to work hard, whatever that takes. Now sometimes it 
could be imaginative, exciting teaching and sometimes it could be drill. Whatever it takes to 
get the students to work hard is what works. And it’s very pleasant to be inspired and have 
beautiful clear lectures, but what counts is, “do the students work hard?” If the beautiful 
clear lectures and consistent evaluation gets the students to work hard, then that’s what I’m 
for. It seems to me that’s what the best teachers do — they give clear lectures, well-prepared, 
without mistakes, and they give straightforward tests, sometimes a little tricky, but by and 
large straightforward, which are evaluated and graded consistently and uniformly. Those are 
the things that seem to me to work. 

Below we restructure Lecturer 2’s analysis, elaborate it with findings from our interviews, and' 
present it from the standpoint of students. 

The DS approach requires that students: 

• Attend lectures (three 50-minute periods/week) in which faculty clearly present the principles and 
methods of calculus and work sample problems, keeping errors to a minimum. 

• “Work hard” in order to apply the methods and grasp the principles. Working hard involves: 

• copying down lecture notes; 

• doing homework problems by oneself and checking oneself by referring to the answer key in 
the textbook; 

• reading the textbook (in some cases); 

• attending discussion section (two 50-minute periods/week) where they listen to their instruc- 
tor explain concepts, and watch their instructor do problems which they and other students 
find especially difficult, and compare the instructor’s methods and answers to their own in 
order to get corrections or affirmation; 

• taking timed discussion section quizzes, receiving graded quizzes back and grades; 

• taking exams (timed or untimed, depending on the lecturer), receiving exam comments and 
grades; 

• And watch the instructor do certain problems that many students found difficult. 
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This analysis of “what seems to work” hinges on a particular approach to problem-solving: a student 
must arrive at a point where he or she can duplicate the problem-solving processes that the lecturer, 
instructor, and textbook present. 

WES Workshops: Rely Primarily on Peers and Secondarily on SAs and Instructors to Explore 
Multiple Approaches to Problems 

Drawing on findings from our interviews, we find that the WES approach retains most elements of 
the DS approach but replaces discussion section attendance and the timed quizzes with: 

• participation in workshops (three two-hour periods/week) where students work with one another 
and get hints and suggestions from student assistants and instructors in order to solve “worksheet” 
problems that are difficult enough to allow multiple solutions, intentionally too hard and too 
many to finish, and have no answer key. 

While retaining drill activity (WES students do homework problems), the WES approach replaces the 
DS problem-solving activity with a significantly different approach. It places students in a situation 
where instructors actively communicate that they assume the students are smart by (among other 
things) offering only hints and suggestions. And the WES approach involves students in problems 
that are open-ended (having no single, obvious answer) and difficult enough that they are forced to 
rely on one another — in work groups — to solve them. 

As the WES students quoted in section II.B.l. above make clear, a key effect of working in groups is 
the development of an approach that incorporates multiple ways of solving prob-lems. As they 
explained, the development of this approach is related in part to the process of group problem- 
solving. Through observing their peers utilizing differing approaches the students begin to integrate 
multiple perspectives as part of their problem-solving skills. Importantly, some students also ex- 
plained that their approach of using multiple ways of solving problems is linked to being confronted 
with very difficult problems. 

It is also of note that some students indicated that as a result of WES they were more likely to try 
different approaches when they find themselves at a dead-end with the approach they first tried. For 
example, one student stated that he had become, 

more likely to be able to solve the problem on my own. You know, if the first method 
doesn’t work, then I have found multiple ways of looking at problems. I think that it has 
shown me, like if I try to take the derivative of something, that maybe that is not the solu- 
tion. Maybe I need to multiply the top and the bottom by the conjugate, or maybe I need to 
do substitution or something else. And I look at a lot more different aspects. 

Another student said that, 

Well I think that even if I don’t use the math itself I will be able to apply the way of thinking 
that it teaches you — a logical way of solving problems. It teaches you how to solve prob- 
lems. And even the thinking, boy if I can’t get the solution this way, maybe I need to look at 
it another way. And keep on looking for a different solution. 

In the WES workshop environment, the issue is not how many problems you answer correctly, but 
how well you understand the problem-solving processes and the concepts which the problems entail. 
The fundamental difference between DS and WES approaches to problem-solving explains why some 
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WES students complained during interviews that their WES work didn’t exactly prepare them to 
perform well on the exams. As will be evident from the course outcomes described in Section 
II.C.l., they did perform well on exams. Nonetheless, many felt as if the exams provided the oppor- 
tunity to utilize only some of the understandings and to perform only some of the skills they had 
developed. 

c. Attitude Toward Learning 

Discussion Sections: Accumulate the Requisite Skills, Content, and Credits 

Most DS students held a strongly “acquisitive” attitude toward learning. While an occasional student 
remarked that he or she enjoyed the course, most made it very clear that they were in the course in 
order to “get” the skills, content and credits they needed to continue with their majors. These atti- 
tudes often were expressed in their answers to open-ended questions like “How do you feel about 
calculus now?” or “What will you remember about this course a year from now?” For example, one 
student said, “I have to know it in the future... I figure I’ll end up using almost all the stuff I learned 
in this course in the next two [calculus courses].” Later, this student elaborated on the value of the 
course, as follows: 

S: As far as problem-solving it will help because no matter what way I go, I mean there’s al- 

ways going to be problems ... worth solving. Calc helps you that way, I think. 

I: Has taking calculus changed the way that you think about real problems? 

S: No. I don’t think so. Not that I notice. 

Another student, in response to the question about what he would remember, said, “What will I 
remember about my calc course? Ahh, not that much. Probably, well like right now we’re doing 
interest rates... I’ll remember that... Other than that, probably not all that much information.” A third 
interview dialogue went as follows: 

I: What do you think about calculus now? 

S: I don’t really think too much about it. Well, one of the things that [Lecturer 2] said today 

was, he’s a professional mathematician, he hardly uses any integration formulas or anything 
like that. And one of the professors in engineering, he goes, “Yeah I’ve had three... semes- 
ters of calculus. I don’t use any of it.”... and one of my friends went through here in bio- 
chemistry and he had three semesters of calculus and he hasn’t used any calculus. But I 
mean it’s, it depends on what you’re going into as far as its usefulness. So I’m not going to 
judge it on that, but there are some parts of it that I find interesting. 

Also representative were responses such as this: 

I: How do you feel about calculus now that you’re almost through the semester? 

S: I’m just anxious just to get it over with. I find it pretty irrelevant, so. 

I: You talked about that before quite a bit — about feeling frustrated that you had to take the 

course and... 

S: Yeah, actually I’m probably not going to use this [because I’m] in management now, so I 

wouldn’t have [had] to take this class. 

I: So what’s going to happen next semester? 
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S: I’m all done with math. 



In sum, while some DS students communicated that calculus is useful for developing one’s problem- 
solving skills, most conveyed that its main value is that it helps students get through engineering, 
science, and some business courses. 

We speculate that there is a relationship between these students’ largely utilitarian attitude toward 
calculus and their tendency to describe the course as “too abstract” and irrelevant to their everyday 
lives. Typical of DS students remarks is this one: 

I have a hard time seeing how this kind of stuff relates to much, to everyday life... because it 
is so abstract. And the things you do — I have no idea where you would use them. 

Another student, when asked about what kinds of problems are mathematically interesting, replied: 

You know I’m not really sure. A lot of it when I thought about dropping it was how am I 
going to use calculus in every day life? And I can understand how you use algebra in every- 
day life and to an extent geometry, but I couldn’t really find a reason for having to know the 
tangent line of a curve at any given point. And I suppose it would have, for me, it would 
have been better to make it relevant to everyday life ... calculus, the purpose of taking it 
seemed to be to get to another point, as a requirement or as a prerequisite or something like 
that. 

It appears that these students’ tendency to view the course as a prerequisite to get out of the way also 
affected their ability to use the ideas in the course as a way of understanding the world in a different 
way. 

WES Workshops: Pleasure in Connecting Calculus to Real Life and Other Disciplines, and Ap- 
preciation for the Beauty of Calculus 

By the end of the semester, WES students held notably different attitudes toward calculus than the DS 
students. WES students were generally more engaged in and appreciative of calculus. They also 
tended to see applications for calculus within their everyday lives. For example, one student, when 
asked how she felt about calculus now that the course was almost over, responded: 

I’m really excited about it, because you know all this math that we’ve done before, there’s 
all these little things and movies and things and people say, “You are never going to use this 
math later.” And I think that basically everything that I’ve done in math, ever since kinder- 
garten, has been working up to what we’re doing right now. I mean we’re having problems 
in books that you can see being solved in the real world. And actually in my chemistry class 
I’m using this stuff faster than I’m getting it in math. I’ve done beyond what we’ve done in 
this class in chemistry, and these [principles] are being used to solve real physical problems, 
and that is so exciting for me. I’m just like, if I can apply something physically, that’s the 
best for me — I just love it. It’s so useful! I’m just like, “Everyone should be required to 
take a semester of calculus.” 

While not all WES students expressed this much enthusiasm, most at least were interested in under- 
standing the deeper workings of calculus. They had gone beyond simply translating word problems 
into mathematical formulations. The following interaction, which took place when an interviewer 
asked a student what he will remember about WES, illustrates these students’ “why” questions. 
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S: [I will] remember, like, learning things that seemed like completely impossible. 

X; Tell me more, that’s interesting. What do you mean by that? 

S: Like, I’ll give an example. Like, today, trying to see a correlation between arcsin and the 

derivative of arcsin — they look completely different. We have arcsin, and the derivative is 
like one over one plus x squared and you just can’t plug it into a derivative thing and you 
can’t just do a power rule, and I remember something like that that seemed impossible, being 

made simple. Things like that. 

I: Is that exciting? 

S: Yah, it is. It’s exciting, because it is like, “God, I can’t do that. It is impossible. And eve- 

rybody kept saying, “You just have to memorize those. You just have to know those. You 
are just supposed to know those.” And then [the senior peer] came over and he was like 
“those are really simple.” And we’re like, “Cool.” And he went through and he taught us 
those and now, now, I feel like I can know them all. If anybody said arcsin or arccos or arc- 
tan, derive the derivative of it or prove it, I would have no problem with that. I remember 
before that, it just seemed like something foreign. And I like those connections with some- 
thing that seemed completely foreign or impossible being made easy. 
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C. Learning Outcomes 

In Section B above, we discussed at length findings based on qualitative research pertaining to the 
learning process as experienced by students in WES workshops and discussion sections. Readers may 
wonder if the apparent improvements associated with the WES Program described above are related 
to improvements in achievement as measured on standard calculus exams. Does the WES Program 
result in statistically measurable improvements in GPA and course completion? To answer this 
question, Joel R. Levin, a UW-Madison professor of Educational Psychology, statistically analyzed 
the available data from this 1993-94 pilot WES program. 

1. Learning Outcomes Indicated by Quantitative Data (by Joel R. Levin) 
a. Introduction 

The design and execution of the pilot WES Program did not lend itself to a rigorous, scientifically 
credible evaluation of the current efficacy and future potential of this approach on the UW-Madison 
campus. A number of critical factors (some of which are pinpointed later in this section) contribute 
to the tentativeness of causal connections made here. Among these, an obvious factor is the nonran- 
dom manner in which students were selected and assigned to the WES and non- WES sections. That 
is, interpretation of the results — and, more specifically, attribution of any observed effects to the 
WES program per se — is complicated by potential selection biases associated with participation in the 
WES program. It cannot simply be assumed that WES and non- WES students were comparable in 
their precollege “background characteristics” (including both intellectual and attitudinal/motivational 
characteristics) at the beginning of the project. To deal with the noncomparability issue, in the 
following quantitative comparisons of students’ course achievement, adjustments (through analysis of 
covariance) were made to control statistically for differences in WES and non- WES students’ rele- 
vant precollege intellectual characteristics. The same adjustments, unfortunately, could not be made 
for potential differences in students’ entering attitudinal/motivational characteristics, in that no 
information related to such characteristics was collected or was readily available. 

In addition, there is a pervasive problem in educational intervention research, referred to in various 
ways as the “units of analysis” problem (Levin 1985, Levin 1992). The problem can be described as 
a mismatch between the number of independent entities to which an instructional intervention is 
applied and the number of entities on which a statistical analysis, assessing the effect of that interven- 
tion, is based. Such a mismatch is known to have both methodological and statistical consequences 
that are sufficiently serious to compromise meaningful interpretation of an intervention’s effect. 
During the first year of the WES project, students in both the Math 221 (Semester 1) and Math 222 
(Semester 2) courses received the primary academic component of the WES intervention in the 
context of two workshops that replaced discussion groups. The WES instructional philosophy 
encouraged continual student interaction and small-group problem solving during these workshop 
meetings. Because each workshop group was associated with its own unique instructional experi- 
ences — including the just-noted group work aspect — individual students cannot properly be regarded 
as the independent units of investigation. Rather, the workshop sections should be regarded as the 
independent units to which the WES intervention was administered. 

Consequently, most analyses reported below are based on two different approaches, hereafter referred 
to as “student-level” and “section-level” analyses: 



erlc®**®™ 



m Final Report, p. 34 




• Student-level analyses represent the traditional approach, wherein the umts-of-analysis problem 
is ignored and the data are aggregated at the level of individual students. In situations where true 
independence among intervention-administered students exists, student-level analyses are both 
appropriate and statistically advantageous. In the present situation, however, where an assump- 
tion of inter-student independence is patently violated, this analysis strategy is known to be 
unjustifiably liberal— and, therefore, inappropriate— from a legitimate statistical inference 

perspective. 

• The section-level approach, although generally more conservative with respect to statistical 
inference, is deemed more defensible for the present quantitative evaluation. Although the results 
from both approaches are presented here, it is recommended that the reader be persuaded more by 
the results from the section-level analyses than by the student-level analyses whenever statistical 
conclusions based on the two approaches differ. 

Unless otherwise stated, the nominal familywise Type I error probability associated with each analy- 
sis was set equal to .05. 

1) Dependent Measures 

The primary outcome measure investigated in the quantitative analyses was the end-of-course grade 
awarded in Math 221 (Fall 1993) and Math 222 (Spring 1994), as represented by a numerical score 
ranging from 0.00 (F) to 4.00 (A). In one set of analyses— referred to as the “course completer 
analysis — students with end-of-course reported grades of Drop, Withdraw, and Incomplete were not 
included. Not including such “no grade” students, however, could bias the results either in favor of 
WES or against it, depending on the comparative proportions of noncompleters in the WES and non- 
WES sections— a methodological issue known more formally as differential attrition. To address the 
potential differential attrition problem, therefore, in a second set of analyses referred to as the 
“estimated grades” analysis— all students were included, with those in the three “no grade” categories 
given estimated (or approximate) grades of 0.5 (midway between a D and F). Although a grade of 
0.5 may be a seemingly pessimistic estimate for some students (e.g., students who were making good 
progress in the course but who were forced to withdraw from school for unrelated personal reasons), 
it was nonetheless deemed a reasonable compromise in order to circumvent the potential differential 
attrition problem and to retain all students with recorded end-of-course transcript information in the 

analyses. 

In the tables that follow, “no grade” percentages are also reported for the reader’s benefit. 



2) Comparison Groups 

For both Math 221 and Math 222, three major instructional categories were created and incorporated 
into the student-level analyses: (1) students in the WES Program; (2) students who were enrolled in 
the same large-group lectures (i.e., had the same course instructors) as WES students, but who 
participated in traditional discussion sections (Comparison); and (3) students who were enrolled in 
different large-group lectures (i.e., had different course instructors) than WES students and who 
participated in traditional discussion sections (Other). For the section-level analyses, the two WES 
workshop sections were compared with all non- WES traditional discussion sections. 
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3) Background Control Variables 

Precollege background data available from students’ admissions files (gender, ethnic classification, 
high school GPA and rank in class, college aptitude scores, UW mathematics placement scores, etc.) 
were considered as potential control variables to include in the analyses. To be so considered, a 
variable: (a) had to be at least a moderate predictor of achievement (i.e., end-of-course grades) in 
Math 221 and Math 222, as determined from both the 1993-94 cohort (excluding WES students) and 
the four prior year cohorts (1989-90 to 1992-93); and (b) had to be based on a sufficiently large 
sample size. With regard to the latter criterion, for example, SAT scores were found to correlate 
moderately with Math 221/222; however, only about one-third of the sample had SAT scores in 
comparison to more than 80% with ACT scores. 

Several multiple-regression analyses determined that: (a) two precollege measures collectively 
did a reasonable job of predicting Math 221/222 grades; and (b) the inclusion of additional precol- 
lege measures did not improve that prediction. The two included measures were high school per- 
centile rank in class (computed from the rank-in-class information on students’ transcripts, to account 
for differing high school class sizes) and score on the UW gamma mathematics placement test. In 
the five Math 221/222 cohorts (all calculus takers for academic years 1989-90 through 1993-94), 
these two measures consistently predicted end-of-course grades to a moderate degree, yielding mul- 
tiple correlations typically greater than .40 and .30 for Math 221 and Math 222, respectively. Adding 
other thought-to-be relevant predictors (e.g., ACT scores, ethnic category) to high school rank and 
UW math placement score did not result in any statistically significant increases in the predictability 
of Math 221/222 achievement. (Obviously, student grades in Math 221 would be expected to be 
potent predictors of their grades in Math 222 and it would have been desirable to include the former 
in analyses of the latter. Unfortunately, the student records accessible to us did not permit a cross 
linkage of mathematics performance in the two courses and, consequently, Math 221 grades could 
not be entertained as a control variable for Math 222.) Based on these multiple-regression analyses, 
high school rank percentile and math placement score constituted two antecedent “control” vari- 
ables in analyses of covariance for Math 221/222 course outcomes, in which the covariate-adjusted 
mean grades of students in the three instructional categories (WES, Comparison, and Other) were 
compared. 

b. Math 221: End-of-Course Grades 

1) Complete Sample: All Usable Records (Table 221-A) 

Course-completer Analysis 

Relevant means for the three instructional categories (along with the sample sizes on which those 
means are based) are presented in the upper portion of Table 221-A. The student-level analysis of 
course-completer adjusted mean grades revealed a signficant difference among the three conditions, 
with subsequent comparisons revealing that the adjusted mean grade of WES students was statisti- 
cally higher than that of students in each of the two traditional sections. (Neither here nor in most of 
the analyses that follow was there a statistically reliable difference between the two non- WES student 
categories, Comparison and Other.) WES students’ adjusted mean grades are between .5 and .6 of a 
grade point more than students in the two other categories. At the bottom of the course-completer 
student-level portion of Table 221-A is shown the complete grade distribution of students by the three 
instructional categories. Apparent here is a higher percentage of As and ABs and a lower percentage 
of Ds and Fs among WES students (46% and 6%, respectively), compared to students in the two 
other categories (about 28% and 16%). Also apparent is the higher “no grade” percentages among 
non- WES students. 
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The superior performance of students in WES sections, indicated in the student-level analysis, is also 
supported by the more rigorous section-level analyses. In those analyses, the adjusted mean grades of 
each of the 66 Math 221 discussion sections comprised the units that were compared. For course 
completers, the two WES sections ranked 1 and 14, which represents a nonchance difference favonng 

WES students, p = .023. 



Table 221 -A 

Total Sample: All Usable Records 



1 . Course-completer Analysis 







111 Comparison 


WES 


: : uiner 








Sh <n = 


599)1 




35) 111: 


i (N = 917) 


Hiah School Rank Percentile 


86.34 


(523) 


86.82 


(33) 


85.57 


(763) 


UW Math Placement 




734.69 


(522) 


734.74 


(34) 


742.68 


(792) 


Grade 




2.55 


(478) 


3.13 


(31) 


2.57 


(784) 


Adjusted Grade 




2.59 


(405) 


3.15 


(28) 


2.56 


(616) 




mill 


AB 


B 


| BC 


■ c ill 


• D, F • 


No Grade 


Comparison 


16% 


11% 


15% 


9% 


13% 


16% 


20% 


WES 


29% 


17% 


14% 


17% 


6% 


6% 


11% 


Other 


18% 


10% 


18% 


6% 


18% 


16% 


15% 



h Section-level fusing mean adjusted grades) 

WES sections ranked 1 and 1 4 out of 66 (p = .023) 



2. Estimated Grades Analysis 
a Student-level 

WES students received higher mean adjusted grades than non-WES students. 
b. Section-level 

WES sections ranked 2 and 13 out of 66 (p = .023). 



Estimated Grades Analysis 

The estimated grades analysis based on all students (awarding estimated grades of .50 for noncomple- 
ters) at the student level confirmed that WES students received higher mean adjusted grades than did 
students in each of the other two categories. Similarly, estimated grades analysis at the section level 
revealed that WES sections ranked 2 and 13, p = .023. 

Besides these analyses, three additional sets of more fine-grained analyses were conducted at both 
student and section levels. These analyses looked separately at different subgroups of students, as 
defined by their year of matriculation, gender, and ethnic classification. Each subgroup analysis is 
discussed in turn. 
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2) By Matriculation Category (T able 221-B) 

All but 2 of the 35 WES students taking Math 221 had matriculated (entered UW-Madison) in either 
the summer or fall of 1993, while 25% of the non- WES students had not. That is, WES students 
were almost exclusively first-semester students, while many non-WES students were not. For a 
number of reasons, first-semester students taking Math 221 would be expected to outperform non- 
first-semester students taking that course. Foremost is that non-first-semester students likely would 
have had to take one or more preparatory college math courses to acquire the calculus-prerequisite 
skills that first- semester Math 221 course takers already had. To control for that potentially con- 
founding variable in the previous analyses, comparisons of WES and non- WES students were re- 
peated, using only summer/fall 1993 matriculants. Although the complete data are summarized in 
Table 221-B, a comparable analysis was not conducted for pre-summer 1993 matriculants because of 
insufficient numbers in the WES category. The non- WES precollege and course achievement data in 
Table 221-B clearly supports the above assumption, that summer/fall 1993 matriculants are more 
successful with respect to the included academic measures than are pre-summer 1993 matriculants. 
Thus, it is important to take matriculation category into account when comparing WES and non- WES 
students. 

Course-completer Analysis 

The course-completer, student-level figures in Table 221-B for summer/fall 1993 matriculants 
parallel quite closely the figures in Table 221-A for the complete sample, as well as the statistical 
conclusions based on those data. Moreover, in the section-level analysis, the two WES discussion 
sections again ranked 1 and 14, p = .023, for students who had completed the course. 

Estimated Grades Analysis 

When noncompleters with estimated grades were included, student-level analyses confirmed the 
advantage for summer/fall 1993 WES students. In addition, in the section-level analysis, the two 
summer/fall 1993 WES discussion sections ranked 2 and 15, p = .030. 

3) By Gender Category (T, able 221-Q 
Course-completer Analysis 

Adjusted mean grades for males and females are presented in Table 221-C. Again the pattern seems 
to be consistent with that observed in the main analyses, with WES students exhibiting an advantage 
of about .7 and .35 of an adjusted grade point in the male and female samples, respectively. Course- 
completer, student-level statistical analysis revealed, however, that the advantage was statistically 
significant only in the male sample. The conclusion was similarly confirmed in the course completer 
section-level analyses: (a) in the male sample the two WES sections’ adjusted mean grades ranked 1 
and 7 out of 66, p = .006, and (b) in the female sample the WES sections ranked only 5 and 43.5 out 
of 66, p > .20. Thus, males in the Math 221 WES sections performed statistically better than their 
non- WES counterparts, whereas females in those same WES sections did not. 

Estimated Grades Analysis 

With respect to estimated grades, the student-level statistical analysis revealed that the WES advan- 
tage was statistically significant only in the male sample. For the section-level anal-sis: (a) in the 
male sample the two WES sections’ adjusted mean grades ranked 1 and 10, p = .009, and (b) in the 
female sample the WES sections ranked only 24 and 33, p > .20. 
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Table 221 -B 






By Matriculation Category 




1 . Course-completer Analysis 








n. Student-level 








Summer/Fall 1993 








Comparison 


,.TC ; WES - < 




Other 


N = 454) 


(N = 33) 




v; (N * 677) 


High School Rank Percentile 88.24 (41 2) 


87.29 


(32) 


87.34 (575) 


UW Math Placement 755.89 (436) 


738.46 


(33) 


766.14 (638) 


Grade 2.67 (383) 


3.14 


(29) 


2.72 (615) 


Adjusted Grade 2.68 (350) 


3.28 


(28) 


2.67 (516) 


No Grade % (1, DR, WD) 1 6% 


12% 




9% 


Pre-summer 1993 








Comparison : 


i® wes m 




•:•••• Other 


' : > M 


immam 




I (N = 240) • V. 


High School Rank Percentile 79.30 (ill) 


71.78 


* 0) v :: 


80.18 (188) 


UW Math Placement 627.20 (86) 


612.00 


0) 


645.45 (154) 


Grade 2.10 (95) 


3.00 


(2) 


2.00 (169) 


Adjusted Grade 


— 






No Grade % (1, DR, WD) 35% 


0% 




30% 


b. Section-level (mean adjusted aradesi 








For summerAall 1 993 matriculants, WES sections ranked 1 and 1 4 out of 66 (fi 


= .023). 


2. Estimated Grades Analysis 








a. Student-level 








For summerAall 1 993 matriculants, WES students received higher mean adjusted grades than their 


non-WES counterparts. 








b. Section-level 








For summerAall 1 993 matriculants, WES sections ranked 2 and 1 5 out of 66 (e 


= .030). 
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1 . Course-completer Analysis 
a. Student-level 
MALE STUDENTS 


Table 221 -C 
By Gender Category 






if Comparison 


•M WES 


llllilliother £ Iffy© TMM'Mti 




(N = 370) 


19) 


fi; (N = 612) 


High School Rank Percentile 


83.60 (326) 


83.29 (18) 


84.01 (513) 


UW Math Placement 


736.36 (322) 


735.84 (19) 


744.04 (527) 


Grade 


2.42 (302) 


3.06 (18) 


2.49 (526) 


Adjusted Grade 


2.54 (256) 


3.23 (1 7> 


2.52 (418) 


No Grade % (1, DR, WD) 


18% 


5% 


14% 


FEMALE STUDENTS 










\V$'<N^22S0i VJ: 


■ 


^ (N = 305) 


High School Rank Percentile 


90.88 (197) 


91.05 (15) 


88.78 (250) 


UW Math Placement 


732.01 (200) 


733.33 (15) 


739.96 (265) 


Grade 


2.79 (176) 


3.23 (13) 


2.72 (258) 


Adjusted Grade 


2.69 (149) 


3.03 (11) 


2.66 (198) 


No Grade % (1, DR, WD) 


23% 


19% 


15% 


b. Section-level ('mean adjusted aradesl 






For men, WES sections ranked 1 and 7 out of 66 (p = 


.006). 




For women, WES sections ranked 5 and 43.5 out of 66 (p > .20). 




2. Estimated Grades Analysis 








a. Student-level 








There was an advantage for male WES students relative to both categories of non-WES male students. 


b. Section-level 








For men, WES sections ranked 1 and 1 0 out of 66 (p 


= .009). 




For women, WES sections ranked 24 and 33 out of 66 (p> .20). 
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Table 221 -D 
By Ethnic Classification 

1 . Course-completer Analysis 
a. Student-level 

AFRICAN AMERICAN/NATIVE AMERICAN/HISPANIC AMERICAN 

Comparison WES . Other 





. (N 


=• 13 ) 


cm 


16 ) l 


• ill <Ni 


23) r 


High School Rank Percentile 


71.21 


(12) 


85.25 


(15) 


81.78 


(14) 


UW Math Placement 


663.88 


(8) 


735.93 


(15) 


719.75 


(12) 


Grade 


1.89 


(9) 


3.03 


(15) 


2.06 


(16) 


Adjusted Grade 


2.84 


(6) 


3.10 


(13) 


2.36 


(9) 


No Grade Percent (1, DR, WD) 


31% 


6% 


30% 








ASIAN AMERICAN 
















iiip 


= 42) 


ilNi 


■ 


lillllll 




High School Rank Percentile 


87.32 


(32) 


76.51 


(5) 


83.09 


(51) 


UW Math Placement 


720.03 


(36) 


686.00 


(6) 


761 .96 


(78) 


Grade 


2.54 


(37) 


2.88 


(4) 


2.84 


(83) 


Adjusted Grade 


2.30 


(26) 


3.66 


(3) 


2.63 


(38) 


No Grade Percent (1, DR, WD) 


12% 


33% 


12% 


(11) 






CAUCASIAN AMERICAN 
















i wxm 


1535^:1111 


■«!■» 


WKKKM 


<N = 


779) 


High School Rank Percentile 


86.66 


(476) 


92.59 


(13) 


85.80 


(691) 


UW Math Placement 


736.45 


(472) 


755.85 


(13) 


739.61 


(686) 


Grade 


2.56 


(424) 


3.33 


(12) 


2.54 


(668) 


Adjusted Grade 


2.60 


(371) 


3.07 


(12) 


2.56 


(565) 


No Grade Percent (1, DR, WD) 


21% 


(111) 


8% 




14% 


(HI) 



b. Section-level fmean adjusted grades') 

For Caucasian-Americans. WES sections ranked 1 and 1 8 out of 66 (p = .038). 
Insufficient data to conduct section-level analyses for remaining two ethnic classes. 



2. Estimated Grades Analysis 

a. Student-level 

There were no instructional differences within any ethnic classification. 

b. Section-level 

For Caucasian-Americans, WES sections ranked 2 and 1 1 out of 66 (p = .01 2). 
Insufficient data to conduct section-level analyses for remaining two ethnic classes. 



O 
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4) By Ethnic Classification (Table 221-D) 

To maintain reasonable sample sizes within the WES category, three ethnic classifications were 
made, representing: (1) combined African American, Hispanic, and Native American students 
(i.e., targeted minorities on the UW-Madison campus); (2) Asian American students; and 
(3) Caucasian students. Although we know that all of the Asian American students in the WES 
Program were members of underrepresented (“targeted”) ethnic groups, we cannot say the same 
for non- WES students. To permit descriptive comparisons here, a single (undifferentiated) Asian 
American classification was used. The data, according to these three ethnic classifications, are 
summarized in Table 221-D. 

Course-completer Analysis 

In the student-level analysis, the descriptive pattern of adjusted mean grade differences within each 
ethnic category is similar to what has already been seen (i.e., WES greater than both Comparison and 
Other). Despite some seemingly sizable adjusted mean differences between WES and non-WES 
students, however, there were no statistically significant effects within any of the separate ethnic 
categories, attributable in part to the reduced statistical power resulting from the ethnic classification 
breakdowns. Similarly, although it is tempting to conclude from Table 221-D that that there was a 
greater percentage of “no grades” in the non-WES than in the WES sections, those differences are not 
statistically significant. In fact, the only nonchance difference in “no grade” percentages involves 
Comparison and Other students in the Caucasian sample. The course-completer section-level analy- 
ses did yield a significant difference in the Caucasian sample, however, in favor of the WES sections: 
The two WES sections’ adjusted mean grades ranked 1 and 18 out of 66, p = .038 in the course 
completer analysis. There were insufficient data to conduct comparable section-level analyses for the 
other two ethnic classifications (targeted minorities and Asian American). Thus, although the 
achievement of targeted minorities is of considerable concern in evaluating the success of the WES 
program, the 1993-94 design and implementation did not allow for a reasonable quantitative assess- 
ment of that concern. 

Estimated Grades Analysis 

As with the course-completer analysis, at the student level there were no statistically significant 
instructional effects within any of the separate ethnic categories. As for the section-level course- 
completer analysis of estimated grades, a significant difference was found in the Caucasian sample: 
The two WES sections’ adjusted mean grades ranked 2 and 1 1, p = .012. There were insufficient data 
to conduct comparable section-level analyses within the other two ethnic classifications (targeted 
minorities and Asian American). 

c. Math 222: End-of-course Grades 

Although the end-of-course grades in Math 22 1 were generally favorable for the WES program, that 
favorability was sporadic, tending to favor WES students overall, and in the subgroup analyses, only 
when restricted to first- semester students, males, and Caucasian Americans. For Math 222, however, 
the end-of-course advantages for WES students were not sporadic, but pervasive and consistent. 
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Table 222-A 

Total Sample: All Usable Records 

1 . Course-completer Analysis 

a. Student-level 

Comparison \'h ;• WES j v • "V Other 











mm <n= 


35) 


(552) 






High School Rank Percentile 


87.78 


(345) 


85.94 


(32) 


86.92 


(462) 




UW Math Placement 




759.50 


(371) 


737.19 


(31) 


759.89 


(477) 




Grade 




2.65 


(327) 


3.10 


(35) 


2.44 


(470) 




Adjusted Grade 




2.58 


(272) 


3.26 


(29) 


2.43 


(385) 






A T;' 


AB 


HHH 




. 'C 


d, f 


No Grade 


Comparison 


14% 


12% 


15% 


15% 


13% 


11% 


21% 


WES 


37% 


14% 


17% 


11% 


14% 


6% 




0% 


Other 


14% 


7% 


20% 


6% 


24% 


16% 




15% 



b. Section-level (mean adjusted aradesi 
WES sections ranked 1 and 2 out ot 45 (p < .001 ). 

2. Estimated Grades Analysis 

a. Student-level 

WES students received statistically higher mean adjusted grades than non-WES students. 

b. Section-level 

WES sections ranked 1 and 2 out of 45 (p < .001). 



1) Complete Sample: All Usable Records (Table 222-A) 

Course-completer Analysis 

Beginning with the complete sample in Table 222-A, we see that WES students’ adjusted mean 
grades were about .7 to .8 of a grade point higher than their non- WES counterparts. In the course- 
completer student-level analysis, WES student grades were statistically higher than in each of the 
non- WES categories. In the lower course-completer student-level portion of Table 222-A are seen 
the comparative grade distributions: more than 50% As and ABs for WES students, compared to less 
than 25% for non- WES students; and 6% Ds and Fs, compared to about 14%. Note, however, the 
comparative noncompletion percentages (which include Drop, Withdraw, and Incomplete): 0% for 
WES compared to about 18% for the two non- WES categories, a statistically significant difference. 
The course-completer section-level analyses produced some particularly striking results. The two 
WES sections ranked 1 and 2 (out of 45 sections) in the course completer analysis of adjusted mean 
grades, p <.001. 
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Estimated Grades Analysis 

In the student-level analysis, WES students were statistically higher than each non- WES category. In 
that all WES students completed the course, they ranked 1 and 2 in the estimated grades analysis, just 
as in the course-completer analysis. 



Table 222-B 

By Matriculation Category 

1. Course-completer Analysis 



a. Student-level 

Summer/Fall 1993 





Comparison 




.0 




Other 






(N = 324) 




(N =31) 


fN=422) : 


High School Rank Percentile 


88.9,4 


(277) 


86.20 


(29) 


88.15 


(371) 


UW Math Placement 


772.97 


(306). 


740.35 


(29) 


775.56 


(394) 


Grade 


2.68 


(272) 


3.15 


(31) 


2.54 


(374) 


Adjusted Grade 


2.54 


(232) 


3.23 


(28) 


2.43 


(329) 


No Grade Percent (1, DR, WD) 


16% 




0% 




11% 




Pre-summer 1993 
















Comparison / 


v'MU Wes 




other 






(N = 89) 




(n=4) 






High School Rank Percentile 


83.08 


(68) 


83.41 


(3) 


81.91 


(91) 


UW Math Placement 


696.09 


(65) 


691 .50 


(2) 


685.49 


(83) 


Grade 


2.53 


(55) 


2.75 


(4) 


2.02 


(96) 


Adjusted Grade 


2.82 


(40) 


4.00 


(1) 


2.43 


(56) 


No Grade Percent (1, DR, WD) 


38% 




0% 




26% 





b. Section-level (mean adjusted grades^ 

For summer/fall 1993 matriculants, WES sections ranked 1 and 2 out of 45 (p <.001 ). 



2. Estimated Grades Analysis 

a. Student-level 

For summer/fall 1 993 matriculants, WES students received higher mean adjusted grades than their non-WES 
counterparts. 

b. Section-level 

For summer/fall 1 993 matriculants, WES sections ranked 1 and 2 out of 45 (p <.001 ). 
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2) By Matriculation Category (T able 222-B) 

Student-level analyses based on summer/fall 1993 matriculants alone (second-semester students) 
yield parallel results (see Table 222- A). As for the Math 221 data, there were insufficient data to 
make comparisons based on pre-summer 1993 matriculants. In the section-level analyses, the two 
WES sections again ranked 1 and 2 on both adjusted mean grade measures (for course completers and 
when estimated grades were included), both ps < .001. 

3) By Gender Category (I able 222-C) 

Recall that in the Math 221 course, WES males statistically outperformed their non-WES peers, 
whereas WES females did not. In contrast, and as may be inferred from the data in Table 222-C, the 
Math 222 student-level course-completer analysis showed a statistical advantage for WES students of 
both genders. 

Course-completer Analysis 

In the student-level analysis, the WES achievement for females (adjusted mean grade of 3.47) is 
especially impressive: (1) in terms of its absolute magnitude (an average grade of AB); (2) relative to 
the two non-WES female categories, where the adjusted mean grade of about 2.54 is almost a full 
grade lower (a grade of BC); and (3) compared even to the WES males, whose adjusted mean grade 
was about .3 units less, at 3.15 (a little more than a B). In the section-level analyses — and as a partial 
mirror image of the Math 221 findings — here it was the female WES students who outperformed 
their non- WES counterparts, with ranks of 1 and 9, p = .020. In contrast, the male WES students did 
not have statistically higher adjusted grades than their non- WES peers in the course completer 
analysis, with ranks of 1 and 19, p = .091. 

Estimated Grades Analysis 

Student-level analyses of estimated grades lead to the same statistical conclusions as in the course- 
completer analysis. In the section-level analyses of estimated grades, the female WES students 
outperformed their non-WES counterparts, with ranks of 2 and 5, p = .009. The male WES students 
also had statistically higher adjusted grades than their non- WES peers: They ranked 1 and 9, p = .016. 
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Table 222-C 
By Gender Category 



1 . Course-completer Analysis 

a. Student-level 

MALE STUDENTS 




r Comparison 


WES Ilii; 


i:;i#;other v 








(N =22) 


(N = 404) 




High School Rank Percentile 


85.99 (236) 


82.84 (21) 


85.57 


(345) 


UW Math Placement 


758.15 (253) 


739.00 (20) 


761.04 


(357) 


Grade 


2.59 (220) 


2.91 (22) 


2.36 


(342) 


Adjusted Grade 


2.58 (181) 


3.15 <19) 


2.41 


(287) 


No Grade Percent (l,DR,WD) 


22% 


0% 


15% 




FEMALE STUDENTS 












(N = 132) 




mm (n= i48)- iiiiii 


High School Rank Percentile 


91.66 (109) 


91.86 (11) 


90.92 


(117) 


UW Math Placement 


762.39 (118) 


733.91 (11) 


756.46 


(120) 


Grade 


2.78 (107) 


3.42 (13) 


2.64 


(128) 


Adjusted Grade 


2.58 (91) 


3.47 (13) 


2.50 


(98) 


No Grade Percent (l,DR,WD) 


19% 


0% 


14% 


(20) 


b. Section-level (mean adjusted aradesl 










For men, WES sections ranked 1 and 1 9 out of 45, £> = .091 , 


, which is not statistically significant 


at the chosen 



significance level of .05. 

For women, WES sections ranked 1 and 9 out of 45, £ = .020. 



2. Estimated Grades Analysis 

a. Student-level 

For men, there was an advantage for WES students relative to both categories of non-WES students. 
For women, there was an advantage for WES students relative to both categories of non-WES students. 

b. Section-level 

Men in WES sections ranked 1 and 9 out of 45, £ = .01 6. 

Women in WES sections ranked 2 and 5, £ = .009. 
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Table 222-D 
By Ethnic Classification 



1 . Course-completer Analysis 
a. Student-level 





Comparison 


WES 


Other 


AFRICAN AMERICAN/NATIVE AMERICAN/HISPANIC AMERICAN 

(N = 1 5) 


(N = 


14) 


(N = 


ID 


High School Rank Percentile 


85.48 (11) 


82.93 


(13) 


87.35 


(9) 


UW Math Placement 


710.91 (11) 


741.64 


(ID 


751.71 


(7) 


Grade 


2.67 (9) 


3.00 


(14) 


2.50 


(10) 


Adjusted Grade 


2.39 (6) 


3.32 


(10) 


2.42 


(7) 


No Grade Percent (1, DR, WD) 


40% 


0% 


(0) 


9% 




ASIAN AMERICAN 




(N = 52) 


(N = 


5) 


(N = 


44) 


High School Rank Percentile 


84.32 (30) 


76.16 


(4) 


83.20 


(18) 


UW Math Placement 


770.41 (42) 


671.60 


(5) 


796.15 


(26) 


Grade 


2.68 (41) 


2.40 


(5) 


2.63 


(31) 


Adjusted Grade 


2.44 (23) 


2.96 


(4) 


2.70 


(11) 


No Grade Percent (1, DR, WD) 


21% 


0% 




30% 




CAUCASIAN AMERICAN 




(N = 330) 


(N = 


16) 


(N = 


485) 


High School Rank Percentile 


88.17 (300) 


91.16 


(15) 


87.04 


(434) 


UW Math Placement 


758.60 (308) 


755.80 


(15) 


757.44 


(439) 


Grade 


2.65 (265) 


3.41 


(16) 


2.41 


(421) 


Adjusted Grade 


2.60 (240) 


3.29 


(15) 


2.42 


(366) 


No Grade Percent (1, DR, WD) 


20% 


0% 




13% 


(64) 



b. Section-level fmean adjusted aradesi 

For Asian Americans, WES sections ranked 2 and 4 out of 23 sections with usable data, p = .024. 

For Caucasian Americans, WES sections ranked 1 and 2 out of 45, p < .001 . 

There were insufficient data to conduct section-level analyses for the African American/Hispanic/Native American 
classification. 

2. Estimated Grades Analysis 

a. Student-level 

For Caucasian Americans, WES students outperformed both groups of non-WES students. 

In addition, non-WES Comparison students outperformed non-WES Other students. 

For students in the combined African American/Hispanic/Native American classification, WES students outperformed 
non-WES Comparison students. 

b. Section-level 

For Asian Americans, WES sections ranked 1 and 3 out of 26, f> = .006. 

For Caucasian Americans, WES sections ranked 1 and 2 out of 45, p < .001 . 

There were insufficient data to conduct section-level analyses for the African American/Hispanic/Native American 
classification. 
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4) By Ethnic Classification (Table 222-D) 

Math 222 data by ethnic classification are presented in Table 222-D. 

Course-completer Analysis 

In contrast to the Math 221 student-level analyses based on course-completing students, where no 
differences among instructional categories were found within any of the individual ethnic classifica- 
tions, significant student-level differences were detected in favor of WES over non- WES in the Math 
222 data set. Am ong Caucasian Americans, WES course-completing students had higher adjusted 
mean grades than did either course-completing Comparison or Other students. In addition, the 
adjusted mean grades of Comparison students were higher than those of Other students. 

Section-level analyses could not be performed in the African American/Native American/ Hispanic 
American sample, due to small sample size. In both Asian and Caucasian samples, however, WES 
students received significantly higher adjusted mean grades. For course completers, the Asian section 
ranks were 2 and 4 (out of 23 sections with Asian usable data), p = .024; and the Caucasian section 
ranks were 1 and 2 (out of 45), p <.001. 

Estimated Grades Analysis 

In the estimated grades analysis, the above-noted advantage of WES students was detected in the 
Caucasian American sample again. Moreover, a similar WES advantage (relative to Comparison 
students) emerged in the African American/Hispanic American/Native American sample. In the 
estimated grades section-level analysis, the Asian ranks were 1 and 3 (out of 26), p = .006, and the 
Caucasian ranks were again 1 and 2, p < .001. 
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2. Learning Outcomes Indicated by Qualitative Data 

This short section presents conclusions about the learning outcomes for WES students which are 
indicated by the qualitative data gathered and analyzed for this evaluation project. First we consider 
the degree to which the learning outcomes intended by the WES Instructors were achieved. Then 
highlight learning outcomes achieved which were not intended by the WES Instructors. 

a. Outcomes Intended by Instructors 

As stated at the outset, the WES instructors’ goals for their students were that they: 

• learn the calculus at a sufficiently deep level that they feel secure in the knowledge that they are 
not just “going through the motions,” plugging numbers into formulas; 

• cultivate a capacity to use multiple ways to solve problems; 

• develop their ability to rely on themselves to formulate and solve problems. 

This evaluation study presents two types of indicators of the degree to which learning outcomes were 
achieved: the qualitative analysis posits learning processes as indicators, while the quantitative 
analysis posits course grades (controlled for precollege measures shown to predict grades to a mod- 
erate degree). Upon reviewing the qualitative data analysis (Section H.B.2.), we find that the WES 
workshops: helped students generate their one solutions by interacting with peers and more- 
experienced others; led students to rely primarily on their peers and secondarily on student assistants 
and instructors to explore multiple approaches to problems; and resulted in students making connec- 
tions between calculus and real life and other disciplines, and in appreciating the beauty of calculus. 
It is plausible to conclude that this combination of learning processes was experienced by the WES 
students and is causally related to a high level of achievement of the above three learning outcomes. 
It is of note that the findings reported in the quantitative data analysis (Section II.C.l) corroborate 
this conclusion. Of particular note, the WES students’ Math 222 grades, evaluated on a course- 
completer student-level basis, were statistically higher than those of DS students: WES students’ 
adjusted mean grades were .7 to .8 of a grade point higher than their DS counterparts. Evaluated in 
terms of the more cautious course-completer section-level basis (also using adjusted mean grades), 
the two WES sections ranked first and second out of 45 sections, an occurrence which is extremely 
unlikely to appear by chance. 

The WES Instructors articulated two more outcome goals. They wanted the WES Program to: 

• provide a math experience that attracts more students into math, science, and engineering majors; 
and 

• provide informal academic and personal advising, and both planned and informal socializing 
opportunities to help first-year ethnic minority and rural students adjust to a research university 
environment. 

With respect to the first of these goals, it is too early to know if the WES students will be attracted 
into math, science and engineering majors at a higher rate than DS students. With respect to the 
second goal, the following qualitative findings indicate that the WES Program had a strong positive 
effect on how well most first-year ethnic minority and rural students adjusted to the UW-Madison: 
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• WES work groups are a source of support 

• WES fosters a sense of safety 

• Group work generates a sense of belonging and being special 

The finding that “work groups vary in character and effectiveness” indicates that these effects were 
experienced more by some WES students than others. Not every student thrived. A very small 
percentage (6%) of the African-/Native-/Hispanic-American WES students received no grade (I, DR, 
WD) for Math 221. (Note, however, that 30% of the DS students in these ethnic categories received 
no grade for Math 22 1 .) 

b. Outcomes Not Expected by Instructors 

We highlight here primarily one learning outcome for WES students that was not expressly intended 
by their instructors. This is that the WES experience cultivates a cooperative approach to learning. 

As we have seen, students in discussion sections generally eschewed the idea of group work on 
calculus problems. For example, 

I: When you study do you usually study alone or do you study with other people? 

S: Usually I study alone. 

I: Why is that? 

S: It seems sometimes that you waste more time when you study with others because you’ve 

got to deal with [what] the person doesn’t understand this, maybe that’s good because then 
you’re reviewing it too, but for me that’s frustrating. 

In addition to generally indicating that they felt group work is a waste of time, DS students expressed 
the idea that group work wouldn’t be helpful in the learning process. This issue arose in the inter- 
views when we described the WES program to the DS students and then asked them if they would 
prefer this method of taking calculus. The students often said that they would be interested except 
that they would not want to work in groups and would not want to spend what they assumed would be 
more total time on calculus. For example, one student responded, “I don’t like working in groups. It 
doesn’t force a person to use his own mind. I don’t know, I just think it is easier when you split up 
responsibilities... you’re not thinking as hard yourself.” Another student in the same focus group 
concurred and added that she didn’t leam as much as a result of depending on others: “I also am not 
big on working in groups.... I don’t leam as much really, you depend on someone else.” 

Many DS students expressed the view that one has to straggle alone to gain an understanding of the 
material. They assumed that in groups one doesn’t work hard or that things will come too easily and 
that therefore true individual learning will not take place. Group work was seen not as something that 
facilitates individual learning but as something extraneous to the learning process. Thus it was 
understandable that these students considered it a waste of time. 

By contrast, students in the WES section developed strongly positive attitudes about their experiences 
with group work. For example, the following passage illustrates strongly positive feelings that 
students in one group had about learning to work in a group. 



I RJC ^ Program Final Report, 



t, p. 50 



SI: I can’t even imagine going on to any science or math class and having just a normal discus- 
sion group. 

S2: What am I going to do when I get thrown into a class without my WES and I just have to 
battle it alone? What am I going to dooo? 

SI: I think too, that because we’ve had [WES] too we’ll be able to go on better to work alone. 

Or be able to develop our own groups to work with.... 

I: Do you think that you will go on to work better alone or do you think this experience will 

make you seek out groups? 

S2: Both. 

I: To try to replicate this in your own way? 

S2: Both. Definitely both. I think that when I work alone it will bring up my own expectations of 
myself and that also I will have a better appreciation of working with other people and valu- 
ing what other people think and what other people say. I used to be a lot more of a loner in 
my thinking. I’d be like, “I know what I’m talking about and you don’t know what you’re 
talking about so I don’t care what you say.” But now I think that because of this I will, if I 
work with other people I won’t think that to myself anymore. I’ll think, “I might not under- 
stand what you’re saying or see the relevance of it but at least I’ll treat you like what you’re 
thinking about math, about science, about whatever, might be worthwhile, that it might per- 
tain to me. 

S3: I would say that before I was not the greatest person to work with. And at high school there 
are lots of people at lots of different levels and I think that it would be more difficult. And 
we’re kind of all at the same level. I was a little skeptical about me working in a group, 
whether or not ... sometimes I’m like, this is not really relevant. But now I think you realize 
in a group setting working with a group is going to help you, but it’s only going to help you 
if you listen to what other people are saying and if they listen to you, and you can bounce 
ideas off of each other. 

SI: Sometimes I’m really bad at that. Sometimes I’m like, “No stop, I need to figure this out.” 
It’s like I said I have to figure out what I’m doing before I can listen to what you’re doing. 

I: It’s almost like you are saying that working in a group is something that you have to kind of 

learn how to do. 

S2: It’s a skill, definitely.... 

Many WES students expressed that they felt an initial trepidation about group work either because of 
a lack of previous experience or as a result of negative high school group experiences. What is 
significant about the above passage is that it makes clear that becoming a productive group is a 
process; the students described their recognition that working in a group is a skill that must be devel- 
oped. A significant outcome of this process is that these students developed an understanding of the 
value of group work in learning calculus. The WES experience taught these students a new method 
of learning. 

WES students indicated that, upon experiencing how cooperative learning provided multiple 
perspectives which helped students — both alone and in their groups — to develop solutions with 
limited help from instructors, they realized that their accustomed solitary work routine was limiting. 
As one student noted, 
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I always feel like when I go [to WES] I’ll be challenged but I’ll come away having learned 
something and understanding, whereas if I’m studying by myself, I know that, yeah, I’ll be 
challenged but I might come away from the session not having solved the problem I went out 
to solve. 

Finally, as the focus group interview excerpt above indicates, students in the WES program trans- 
ferred their experience of working in groups to other settings where they develop groups on their 
own. Some students said that they did this in other classes while some informally studied with others 
or turned to other students for assistance outside of class when otherwise they would have worked on 
their own. 

c. Note on the Hawthorne Effect 

It may be argued, and rightly so, that the WES students’ high levels of performance are partially a 
function of the “Hawthorne Effect,” an improvement in performance attributed to the effect of 
receiving the extra attention associated with participating in an experiment. Simply knowing that 
they were receiving extra attention (including lengthy assessment interviews) as participants in a pilot 
program made the WES students feel special and may have improved their performance. This said, 
we suggest that the Hawthorne Effect accounts for only part of the difference observed between the 
DS and WES sections. For one thing, we observe above (see Sources of Affirmation) that the WES 
approach is structured so that students pay far more attention to what their peers think of them than to 
what their instructors think of them. That most WES students barely noticed us when we came to 
observe them (our presence was more intrusive in the discussion section environments) suggests, by 
extension, that the WES environment also diminishes the influence of evaluators. The students focus 
primarily on their work and each other, and pay relatively little attention to authority figures in the 
environment. For another thing, we propose that the effects of acquiring the kind of group identity 
described by WES (and most distinctly not described by DS) students are not caused by simply 
knowing one is participating in a one-of-a-kind program. Just as the effects of membership in a 
Greek organization or other group with which one identifies would not be substantially changed if 
one knew there were no other such organizations on campus, the effects of belonging to a WES group 
are not likely to be substantially changed because the students know there are no other WES groups 
on campus. 
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III. Implementation Processes and Outcomes for Administrators 

A. From the Observers’ Point of View: Context and Activities for Administrators 

It is common to present a very concise account of program implementation activity. This accords 
with the convention that program implementation activity is, while not unimportant, certainly not 
interesting. Moreover, detailed descriptions of implementation processes often cannot be produced 
even if wanted: in the rush to accomplish this kind of work, detailed records are not kept, and the 
complex panoply of participants and events is forgotten or remembered in a confused fashion. 
Challenging the convention that implementation activity is not worthy of a lengthy account, the 
evaluation researchers kept records of and wrote up this activity. To help readers appreciate the 
contrast between a concise and lengthy accounts, we first present a brief version of WES planning 
and implementation activity, and then a more detailed account. The detailed account provides a 
glimpse of the staff, faculty and administrative cultures in which this program developed — 
intimations of what Clifford Geertz would call a “thick description” (1973). The details provide a 
sense of the complexity of accomplishing what might otherwise appear a simple change. In the 
interest of space, these contrasting accounts only cover the first five months of activity, from Septem- 
ber 1992 through January 1993. Implementation activities that occurred starting in February 1993 are 
presented in bullet form. 

1. Concise Account of the First Five Months’ Planning and Implementation Activity 

Plans for the WES Program began in September 1992 during a discussion following Dr. Treisman’s 
presentation to members of the UW-Madison Mathematics Department. Included in the discussion 
were the Department Chair and Associate Chair, both of whom strongly supported the “emerging 
scholars” idea. Two members of this discussion group, Professors Steven Bauman, Chair of the Math 
Department s minority initiatives committee (MIC), and Michael Bleicher, took responsibility for 
planning and implementing the program. During fall 1992 they developed a proposal for the 
“Wisconsin Emerging Scholars” Program. In December 1992, the 80 person Department of Mathe- 
matics voted to approve the Program on a pilot basis. Early in 1993, the Department sought and 
obtained Divisional Committee approval to allow WES students to take two-credit “undergraduate 
research” courses in conjuction with the regular five-credit calculus courses, for a total of seven 
calculus-related credits per term. When Dr. Bauman announced he would be on leave in 1993-94, 
Faculty Associate Melinda Certain agreed to assist Mike Bleicher. 

2. Detailed Account of the First Five Months’ Planning and Implementation Activity, 
a. Unlikely Beginnings 

One strand of this story begins in the summer of 1992 with conversations about math education 
research between Susan Millar, a Penn State higher education researcher, and Uri Treisman, Director 
of the Dana Center for Mathematics and Science Education at UT- Austin and a well-known higher 
education math educator. These conversations led to the idea of a possible video project. Because of 
relevant professional contacts at UW-Madison, S. Millar arranged to meet in Madison with Treisman, 
Jackie McCaffrey, (Director of Special Projects and Associate Executive Director of the UT- Austin 
Dana Center), and three UW-Madison videographers. While in Madison and independent of the 
video project, Treisman proposed presenting a seminar for the Department of Mathematics. Since he 
knew that although the UW-Madison Department of Mathematics was one of the largest research 
departments in the country (66 FTE faculty), its track record for producing female and minority 
mathematics majors was relatively poor. He pursued the idea with Terry Millar, a professor in the 
Math Department and at the time an Acting Associate Dean in the Graduate School. 
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T. Millar first discussed the idea with Steve Bauman, a professor of mathematics who already had 
shown strong interest in minority student education. (Bauman participated in an interinstitutional 
faculty exchange program by spending a semester at Spelman College during 1990-91). Having 
heard Treisman give a talk at a national meeting, Bauman had previously discussed with Math 
Department colleagues the idea of starting a Treisman-style calculus workshop program. However, 
nothing had come of these interactions. After Bauman expressed renewed interest in such a project, 
Millar consulted with the Department Chair, Alex Nagel, about the possibilities of a Treisman semi- 
nar. The Chair agreed, on the condition that Treisman not push a hidden agenda. T. Millar conveyed 
this sentiment to Treisman, Treisman assured him that there would be no hidden agenda, and the 
Chair then gave his consent. In summary, a video-planning meeting arranged by a researcher (S. 
Millar) who was clearly an outsider to the situation led to an invitation to Treisman by a mathematics 
department that otherwise probably would not have so acted. 

To prepare for his seminar, Treisman requested data on minority student performance in introductory 
math courses at UW-Madison. Responding to the request, T. Millar sought math student performance 
data from the university’s administrative data processing center. He requested data on 
“drop/withdrawal status” and “grade earned” for students in precalculus and calculus classes, by 
ethnicity and gender. Obtaining these data required time and patience, as the university was not 
structured to make such data easily available. In fact, it was alleged that previous administrations had 
a policy (unstated) of making such data virtually inaccessible to casual inquiries. First T. Millar 
contacted the Office of Budget Planning and Analysis, which did not have the data but volunteered to 
secure it from the Division of Information Technology (DoIT). After three weeks and complications 
pertaining to the definition of the data constraints, Budget Planning advised T. Millar that he should 
contact DoIT directly if he was to obtain the data before Treisman’s visit. Millar therefore sent a 
memo describing his request to a midlevel DoIT administrator. This administrator then had to seek 
approval from the “owner” of these data, the Registrar s Office, before DoIT could honor the request. 
Shortly before Treisman s visit, T. Millar received the data, which he analyzed, using a desktop 
database manager. He provided the analysis — which showed a poor performance record for minority 
students — to the chair and Treisman just prior to the latter’s arrival. 

Meanwhile, T. Millar also produced a poster for the seminar, which he distributed to all members of 
the Math Department, and to campus administrative units which had shown interest in minority 
student success in the math-based disciplines. Upon learning of the seminar, several members of the 
Math Department expressed misgivings about the Department possibly being pressured to do some- 
thing that did not suit the will of the faculty. 



b. The Treisman Seminar 

Held in the Math Department s spacious faculty and graduate student lounge, the noon seminar 
attracted some eighty people. Most were faculty and students of the Math Department. The 
approximately twenty administrators and faculty from other units across campus represented the 
College of Engineering, the Center for Biology Education, the Office of the Dean of Students, the 
College of Letters and Science, the Chancellor s Office, and the Graduate School. 



At T. Millar’s request, Bauman (pointedly, a faculty member with no administrative “taint”) intro- 
duced Treisman. The introduction was short. The lecture began on a dissonant note when a member 
of the Math Department challenged Treisman on the accuracy of his review of the University’s 
history. This, plus remarks from others that bordered on heckling, placed Treisman in a defensive 
position. According to his colleague from UT-Austin, Treisman responded with uncharacteristically 
critical and challenging remarks about the Department. In particular, he announced that of all the 
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schools in its peer group, this Department clearly had the worst record for minority student perform- 
ance in precalculus and calculus. After quoting from the data T. Millar had provided, he was finally 
able to shift to his main topic: his programs at UC-Berkeley and UT-Austin. Toward the end of his 
presentation, Treisman challenged the Math Department by saying he would not send any of UT- 
Austin’s ethnic minority math students to the UW for graduate school until the Department illustrated 
(in its performance records for undergraduates) that it had begun treating minority students as young 
mathematicians rather than as minority students. After about 50 minutes, Treisman took questions. 
Audience response was positive and animated. 



c. Post-lecture Reaction 

Several mathematicians, (Bauman, Mike Bleicher, Deiter Uhlenbrock, Alex Nagel, and Claire Rider), 
and Al Hampton, the minority coordinator in the College of Engineering, gathered in a small circle to 
ask more particular questions of Treisman and McCaffrey. The questions were detailed, and often 
fielded by McCaffrey, who provided animated responses illustrated with specific examples. During 
the meeting, Treisman issued an invitation to interested mathematicians to visit UT-Austin, where he 
would show them around the Emerging Scholars Program. This meeting concluded after about an 
hour. Afterwards, Bauman took Treisman and McCaffrey out for refreshments. The assumption at 
the time was that Bauman would act as a key person for followup. 

T Millar analyzed audience response to the talk as follows: “You have to understand that this 
interest was very unusual for mathematicians. Mathematicians often reject out of hand talks like this. 
They just walk out shaking their heads.” He reported that the next day, a mathematician accosted him 
in an accusing tone: “Are you responsible for bringing Treisman here!?” Trying not to be defensive, 
Millar answered that he was, upon which his colleague smiled broadly and said, “Thank you! 



d. Initial Planning Activity 

During the next few weeks, Bauman and Bleicher called informal meetings to discuss the possibility 
of starting a workshop calculus program at UW-Madison. Although invited to participate, T. Millar 
chose not to attend, believing that his role as Associate Dean in the Graduate School made his partici- 
pation at that point too heavy handed. Various members of the Department attended a series of four 
or five meetings during October and November, 1992. During this time, Bauman announced that he 
would be on sabbatical in 1993-94, and Bleicher then began taking a leadership role. Other faculty 
involved at this stage included: Don Crow, Josh Chover (just retired), Uhlenbrock (initially in his 
capacity of Associate Chair), Dan Rider, Phil Miles, Bob Wilson (part-time in UW-Extension), and 
C. Rider (mathematician, academic staff). This ad hoc Wisconsin Emerging Scholars (WES) group 
was a subset of the unofficial minorities initiatives committee (MIC) initiated by Bauman. 

In late October, the ad hoc WES group approached T. Millar with a proposal for Graduate School 
funding to send one faculty member to UT-Austin to meet with McCaffrey and observe the 
UT-Austin program. T. Millar sent their proposal to the Chancellor’s Interinstitutional Linkage 
Committee, of which he was a member. The funding was approved and Bleicher made a two-day 
visit to UT-Austin in late October. 



Meanwhile, the informal group grappled with when to initiate their program. Bauman proposed 
spending a year planning the program and taking the time to seek administrative support. Although 
this viewpoint had its proponents, others preferred to offer a pilot program in fall 1993. They feared 
that spending a full year in planning would not increase the likelihood of administrative support and 
that the Department then would have expended much effort needlessly. This second view prevailed, 
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and the group developed a two-part strategy for obtaining financial support immediately. First, 
Bleicher and Bauman agreed to apply to the UW-System Administration Undergraduate Teaching 
Improvement Council (UTIC) for a grant to help support the teaching costs of the proposed program. 
Second, Bleicher and Crow prepared a draft document requesting various resources — including a 
half-time administrator — to initiate the proposed WES Program. They presented this document to 
T. Millar in November. Millar requested additional details, which the committee members sought 
from UT-Austin’s McCaffrey. 

In preparation for the WES group’s next meeting, T. Millar talked to Gary Sandefur (Associate Vice 
Chancellor), Phil Certain (Associate Dean of Letters and Science), and Alex Nagel (Chair of Math). 
All of these individuals supported the WES idea, and agreed to attend planning meetings. However, 
they felt it was premature for the Math Department to ask for administrative support, particularly 
since the Department’s student enrollments had declined recently. Both Nagel and P. Certain also 
expressed concern about who in the Math Department would assume the responsibility to say, “This 
is going to work, and I m going to make sure it works!” Certain was reluctant to co mmi t to the 
program until this kind of leadership emerged. 

During the next WES group meeting, attention focused on how to overcome administrative difficul- 
ties such as obtaining longitudinal data on precalculus students. To help the WES group understand 
how many different people were involved, T. Millar presented a diagram of the administrative units 
that he perceived as relevant to the proposed WES Program. Seeking to change the impression that 
there is a “monolithic bunch of administrators out there,” he explained that the human resources 
needed to make this program work were distributed across various offices around the campus. He 
then suggested that the WES group invite these diverse administrators to a meeting in order to 
establish possibilities for long-term administrative support. Bleicher suggested that T. Millar become 
the key leader. T. Millar declined, saying the leader should be someone who did not have an adminis- 
trative role outside the Department. He then arranged a “rehearsal,” during which the members of the 
WES group developed a detailed agenda and assigned different individuals responsibility for each 
agenda item. During this meeting Bleicher assumed the role of key spokesperson. The rehearsal 
provided the members of the WES group the “extra polish” needed to make a convincing presentation 
to university-level administrators. 

e. First Administrative Meeting 

To this meeting T. Millar invited several members of the Math Department, two Associate Vice 
Chancellors, the College of Engineering’s Assistant Dean for Minority Affairs, two Assistant Vice 
Chancellors, the Graduate School’s Associate Dean for Minority Affairs, the Associate Director of 
the Center for Biology Education, and the College of Letters and Sciences’ Associate Dean for the 
Natural Sciences. All but one of the invited administrators attended, along with five members of the 
Math Department (Bleicher, Bauman, Uhlenbrock, Nagel, and Chover). The meeting went as 
planned, and the invited administrators expressed strong moral support. 

f. First Math Department Vote 

The Math Department Chair placed the proposed pilot WES Program on the agenda of the December 
1992 Department meeting. It was necessary to vote on the pilot program because it would be for 
seven credits, instead of the usual five-credit calculus course. These extra credits required a new 
course number and thus Divisional Committee approval. The Department meeting was adjourned 
before a vote was taken. Since the next Divisional Committee meeting was to be in mid- January, and 
no other Math Department meeting could be held before then, Nagel was forced to send a mail ballot 
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to the Department, an action with which he felt uncomfortable. The ballot tally was: 44 approvals, 3 
abstentions, and 5 disapprovals (two faculty indicated a desire for the issue to be discussed at a 
departmental meeting). Nagel ruled in late December that the pilot program was approved. 

g. Dropping the Ball 

Although the request (for a course with seven credits) was therefore forwarded to the Divisional 
Committee, no one from the Department “prepped” Divisional Committee members about the details, 
not even that member of the Committee who was from the Math Department. This omission had 
nearly catastrophic consequences. During their January meeting, the Divisional Committee inter- 
preted the Departmental proposal as a request for a new course intended solely for minorities. This 
erroneous interpretation was exacerbated by the inability of the Divisional Committee member from 
the Department to provide clarifications (and again, this was not his “fault,” since he had not been 
briefed by the WES Committee). Not surprisingly, the Divisional Committee turned down the 
request. 

When the Divisional Committee action became known, an “emergency” meeting of the WES group 
was called, and the Math Department member of the Divisional Committee also was invited. It was 
at this meeting that the WES group came to understand how they had “dropped the ball.” Profuse 
apologies were offered to the Divisional Committee member for having put him in such an untenable 
position. With this individual’s help, a new strategy emerged that was ultimately successful. Instead 
of a seven-credit course, the Department subsequently requested a new course number with flexible 
credit hours that was intended for undergraduate research (and thus would potentially play a role in 
Department course offerings outside of the WES Program). This proposal ultimately was approved 
by the Divisional Committee. Thus students in WES were to receive the usual five credits for the 
calculus course and two credits for an undergraduate research course. 

h. Another Potential Setback Is Avoided 

By December 1992 it was generally agreed that Bleicher would play the role of the WES Program 
director. Anticipating how much extra work it would be to prepare for the first semester of the WES 
Program, he asked the Chair to arrange release time from one of his spring 1993 courses. Nagel 
forwarded this request to the College Dean’s Office, where it was discussed. From the Deans’ 
perspective, plans for implementing the WES Program were too amorphous to earn their vote for the 
release time. At first, this negative decision discouraged Bleicher. It led him to realize that adminis- 
trators often require detailed plans for an innovative activity before they will release resources for it, 
even though it is not possible for the innovator to produce such plans precisely because the activity 
has no prototype. This potential setback to the WES planning process was resolved, however. The 
course from which Bleicher would have been “released” was an advanced graduate course in his area 
of expertise. He realized that if this course were cancelled to provide release time, he would end up 
teaching it informally anyway. In the end, therefore, he willingly proceeded with the time-consuming 
work of preparing for the WES Program while teaching his regular course load. 

L Details, Details 

In early January, 1993, a Math Department instructor, Melinda Certain, became involved as another 
WES Program leader. She helped Bleicher with the “detail work” required to make the WES group’s 
plans for the fall 1993 pilot WES cohort a reality. In “steady state,” many of these activities could be 
assigned to academic staff and classified personnel. However, such personnel in the Department were 
over-extended, and therefore the Chair made it clea^ tiiat no support of this type could be expected. 
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Below we describe the details to which Bleicher, M. Certain and others attended through February 

1993 . 

• Arranging an option for students to live in a WES "house" in one of the dorms. The details of 
arranging special housing for WES students entailed an initial meeting between the Associate 
Dean of L&S (Phil Certain), an Associate Vice Chancellor (Sandefur), an Associate Dean of the 
Graduate School (T. Millar), and the Director of University Housing (Norm Sundstad). Sunstad 
said it would be no problem to designate the floor of one dorm as a WES “house,” and selected 
for WES the floor directly above the honors house in one dorm. The group discussed how stu- 
dents made housing requests and how officials made assignments. Sunstad noted certain con- 
straints, such as the need to ensure that the floor is handicapped assessable because it is associated 
with a program. He offered to organize a series of subsequent meetings to work out the technical- 
ties of the paperwork and to inform the WES leaders of other programs that have housing pro- 
grams. M. Certain, Bleicher and housing staff met once, and later toured dorms. M. Certain took 
over, the details of implementing the housing arrangements. 

• Providing WES students a classroom of their own. T. Millar began the process of obtaining a 
WES classroom by speaking with another Assistant Vice Chancellor (Akbar Ally). In early 
February, Sandefur assumed responsibility for this space request and investigated how to obtain 
space in two buildings adjacent to the math building. Finally, contacts with the Eberhardt 
Schubert, of the Institutional Space Office, resulted in space tours for Bleicher and Millar, 
Bleicher and Certain, and Bleicher again. The WES leaders finally settled on a spacious and 
pleasant, but unfurnished and poorly-maintained room on the third floor of Bascom Hall, very 
close to the Math Department. 

• Recruiting 

• Arranging for access to DoIT data. In order to determine which accepted students had suffi- 
ciently high ACT or SAT scores to be invited into the Program, it was necessary to get access 
to admissions data. In early January, T. Millar had a phone conversation with Esrold Nurse, 
the Associate Director of Admissions, as a follow up to two mid-December meetings between 
Nurse, Bleicher, and a staff person from DoIT. T. Millar indicated that it would be useful to 
have admissions data available electronically, and Nurse offered to meet with DoIT to hammer 
out details. 

• Arranging to meet potential WES students during Student Orientation (SOAR). The WES lead- 
ers believed they should meet with the students invited to join the Program when they came to 
campus for their summer orientation program (SOAR). To arrange a booth at SOAR, Bleicher 
and T. Millar met with the Mary Rouse (Dean of Students) and Jack Kellesvig (Associate Dean 
of Students), and Bleicher then held additional meetings with Kellesvig. 

• Seeking advice on minority recruitment strategies. The WES leaders learned that A1 Hampton 
and Don Woolston, two Assistant Deans in the College of Engineering, had considerable expe- 
rience recruiting minority students. Accordingly, Bleicher, M. Certain and T. Millar met with 
them to obtain their advice. 
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• Arranging for WES computer needs. T. Millar, Bleicher and M. Certain assessed their com- 
puter facilities, determined that neither Bleicher’s nor M. Certain s computer would run a 
desktop database manager, and decided they should use T. Millar s machine in the Graduate 
School to analyze admissions data. The Graduate Dean, John Wiley, enthusiastically supported 
this idea, and authorized M. Certain to have access to a portion of the Graduate School s com- 
puter network. 

• Deciding on selection criteria. The WES group met to choose the criteria that would be used to 
determine which admitted students should receive invitations to the Program. They decided 
invited students should have ACT math scores greater than 22 and/or SAT math scores greater 
than 550. A DoIT representative indicated that it would be straightforward to select records 
that met these criteria. 

• Obtaining necessary admissions data. Toward the end of January, Nurse arranged a meeting 
with Larry Frisch (an information specialist in DoIT), T. Millar and Bleicher. They determined 
which data elements available on undergraduate applicants would be provided to the WES 
Program. These elements included: self-reported grades in high school core courses; number of 
credits in English, math, social studies, and foreign languages; ACT and SAT scores; high 
school name; and the usual demographic information. T. Millar mentioned that it would be 
ideal to receive the data in electronic form to use in a relational database, and Frisch said this 
was possible. 

Bleicher, Certain, and Millar planned to report at the Math Department’s Undergraduate 
Education Committee meeting on how many and which types of students would be recruited. 
They learned a couple days before the Committee meeting that, because information on ethnic- 
ity was among the data sought, it would be necessary to obtain various authorizations before 
Bleicher and M. Certain could have the data. Thus, more crosscampus coordination was 
required to gain release of the data. The day of the meeting, the data finally was transferred to 
T. Millar, who prepared it in table form, sorted by GPA and math courses. 

• Meeting with the Math Department ’s Undergraduate Education Committee. The Education 
Committee found it unexpected and very interesting that there were 1 63 minority applicants who 
met the criteria determined by the WES group. The Committee was briefed about recent events. 

j. Concluding Comment 

Readers should rest assured that the same level of preparatory activity continued from February 
through August 1993. In the interest of brevity these activities are presented in bulleted form. 

Remaining Planning and Implementation Activity. The implementation activities that took place 
from February 1993 through the beginning of the fall 1993 semester are presented below. 
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• In winter 1993, Drs. Bleicher and Certain visited the UT- Austin Emerging Scholars program to 
observe teaching/leaming patterns in the workshops, obtain sample worksheet problems, and learn 
how to administer this kind of program. This visit was financed by the UW-Madison Interinstitu- 
tional Linkage Committee. 

• In midspring, Drs. Certain and Bleicher, with the assistance of various other Math Department 
and campus administrators, proceeded to: 

• develop a program brochure which they sent to selected first-year students planning to attend 
UW-Madison in fall 1993; 

• complete arrangements for a dorm “house” where WES students could choose to live; 

• obtain grants worth $ 1 8,000 (roughly half each from the UW System Administration and the 
UW-Madison Provost’s Office) to cover costs of secretarial support, recruitment, field trips, 
computer equipment and most other expenses associated with the program; 

• obtain release time support from the College of Letters and Sciences for Dr. Bleicher to 
administer the WES Program and permission for Drs. Bleicher and Certain to act in the roles 
normally played by graduate TAs during 1993-94; and 

• spend many hours selecting and recruiting prospective WES participants. 

• In late spring, Drs. Bleicher and Certain received sufficient response to their invitations to run 
two WES workshops. 

• In early June 1993, the Dean of the College of Letters and Sciences allocated a 12-month, half- 
time project assistantship to assist with the evaluation of the program. (The project assistant 
would be a doctoral student in the UW-Madison Department of Mathematics.) 

• In August, Drs. Certain and Bleicher hired two math majors to act as undergraduate “student 
assistants” in the WES workshops. 

• During the week before classes began, Bleicher and Certain prepared for and held a WES 
reception party for all WES students, Math Department faculty and a large number of campus 
admi nistrators. They sent invitations, made special contacts with WES students to encourage 
them to come, and arranged for refreshments (pizza and soda). A large proportion of both the 
invited faculty and administrators and WES students attended. 

As of fall 1993, the intensity of work dropped off for most participants. However, Bleicher and 
M. Certain, who actually taught the pilot course, experienced an increase in work intensity during the 
fall semester. The instructors found that the workload leveled off during the second semester, spring 
1994. 
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B. Implementation Processes for Administrators 

This two-part section features the implementation processes experienced by administrators involved 
with the WES Program. The first part presents implementation processes as described by admini- 
strators themselves, while the second features the researchers’ analytic generalizations. 

1. From the Participants’ Points of View: Key Implementation Experiences 

a. Department Level 

The above account of the context and activities for administrators indicates that the department-level 
people who implemented the WES Program were: the Math Department’s MIC committee (Bauman, 
C Rider, Crow, Chover, D. Rider, Miles, Wilson); the Chair (Nagel) and Associate Chair 
(Uhlenbrock); the faculty member who became the WES Program director (Bleicher); and the faculty 
affiliate who assisted the director (M. Certain). These Department faculty and administrators tended 
to remark on the remarkable complexity of the process. Most were surprised at the number ot 
campus offices and administrators who needed to be involved. The Program leaders voiced some 
frustration at the many steps that had to be taken and the numerous small setbacks they had encoun- 
tered. They also expressed excitement at their success, and appreciation that they received any 
recognition at all from Department, college and university, and extra-university administrators. 

b. College Level 

The individuals at the college-level who played key roles implementing the WES Program were the 
Associate Dean for the Natural Sciences (P. Certain) of the College of Letters and Sciences. Also 
important was an Assistant Dean of the College of Engineering (Hampton). 

The Associate Dean for the Natural Sciences was cautiously supportive of the WES Program, as 
expressed in the following combination of actions. He provided support for the external evaluation 
work (represented in this report) by providing a graduate project assistantship, encouraging his staff 
to expedite the human consent process, and providing the external researcher access to student 
database records. He chose not to provide faculty release time during spring 1993 when the program 
was being planned, but did agree that a faculty member and faculty associate should act m roles 
normally played by graduate teaching assistants for the first year of the Program. He provided very 
limited additional financial resources (in the form of a new blackboard for the WES classroom). In 
spring 1994, he indirectly supported the program by sponsoring a new “Excellence in Undergraduate 

Education” (EUE) program. 

c. Institution- and Extra-institution level 

A review of the foregoing account of the context and activities for administrators produces the 
following lists of institution and extra-institution-level units which played roles at key points m the 
implementation of the WES Program. As the individuals associated with these units were not inter- 
viewed, their implementation experiences are not represented here. 
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Institution-level offices 

Chancellor’s Office (Associate Vice Chancellors Gary Sandefur, Akbar Ally) 

Division of Information Technology (Larry Frisch) 

Facilities Planning and Management (Eberhard Schubert) 

Graduate School (Associate Dean Terry Millar, Dean John Wiley) 

Office of Admissions (Associate Director Esrold Nurse) 

Office of the Dean of Students (Associate Dean John Kellesvig, Dean Mary Rouse) 

University Housing (Director Norman Sunstad) 

Office of the Registrar (Associate Registrar Herbert Evert) 

Extra-institution-level organizations 

UT- Austin Dana Center (Uri Treisman, Jacqueline McCaffrey) 

UW-System Undergraduate Teaching Improvement Council 

The National Science Foundation (support for the external evaluation study) 

The University of Maryland University College’s Institute for Research on Adult Education 
(support for the external evaluation study) 

2. From the Researchers’ Point of View: Analytic Generalizations 

UW-Madison is a major public university whose primary mission, as perceived by the faculty and 
staff, is research. In light of this context, there was moderate support for the program from all 
administrative levels — departmental, college, institutional and even extra-institutional — during the 
early implementation stages. This said, it is of note that the implementation process was sufficiently 
complex, time-consuming and, at times, precarious that leaders with less tenacity and vision might 
have abandoned the entire effort. Without the combination of formal and informal recognition, some 
financial support, and the personal commitment and tenacity of key leaders, it is unlikely the Program 
would have been implemented. Implementation of a program such as WES requires not only sub- 
stantial time and effort, but also attention to a deceptively complex set of organizational matters. 
Moreover, recognition and support from colleagues and administrators are tenuous and circum- 
scribed. Given these challenges, it is understandable why few faculty succeed in implementing 
innovative teaching programs. 
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C. Implementation Outcomes 

1. Continuation of Pilot-level Program 

In fall 1993, Richard Brualdi, the Math Department Chair appointed in summer 1993, and Mike 
Bleicher submitted a proposal for an Excellence in Undergraduate Education (EUE) grant from the 
College of Letters and Science. The proposal was funded on the condition that the Department accept 
the College’s revisions. The EUE provided the Department sufficient extra funding to support the 
following number of fall 1994 WES workshops: three Math 221 workshops, one Math 222 workshop, 
and one Math 223 (third-semester calculus) workshops, plus three Spring 1995 Math 222 workshops. 
Of the planned 1994-95 workshops, one was set aside for enrollment by pre-engineering students 
only, and was supported by a Technology Reinvestment Program (TRP) grant obtained by the 
College of Engineering’s Engineering Research Center for PlasmaAided Manufacturing. In addition, 
the TRP grant provided some summer support for Professor Bleicher to develop Math 223 worksheet 
problems. As of fall 1994, a sufficiently large number of engineering students sought entry to the 
WES-for-engineers workshops that an additional TRP-funded section was offered, bringing the total 
number of Math 221-222 WES workshops to four. 

2. Plans for Scaleup and Institutionalization 

The informal WES Program Committee continued to meet sporadically during 1993-94 and into the 
fall of 1994. Two fall 1994 meetings were devoted to planning for a Department meeting devoted 
almost exclusively to consideration of the WES Program. The November, 1994 Department meeting 
featured presentations of evaluation data by Susan Millar, Director of the LEAD Center (see Appen- 
dix C), and Joel Levin, a professor from the Department of Educational Psychology. The Department 
Chair then presented data indicating that the net financial cost of the WES Program to the Department 
was very low, in part because each WES student earns 7 math credits per semester, making the WES 
TA student credit load comparable to the DS TA credit load. After an animated and generally very 
positive discussion, the Department voted almost unanimously to recommend to the Executive 
Committee hiring a WES Program Coordinator and supporting the WES Program on a pilot basis 
until 1998-99. During this pilot period, the Program was to scale up very slowly. In the spring of 
1999, the Department would review the accumulated evaluation data (focusing particularly on 
longitudinal outcomes for ethnic minority and women WES students) and decide whether or not to 
permanently institutionalize the Program. 
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IV. Conclusions 

In this two-part section, we respond to the evaluation research questions articulated at the beginning 
of this evaluation study. 

A. Learning Processes and Outcomes for Students: Summary Findings 

We list here s umm ary findings for each research question (presented in bolded italics) which shaped 
the data collection and analysis processes which focused on student learning processes and outcomes. 

1. Overall, do WES students learn calculus better than discussion section students? More 
specifically, 

• What is the role of the worksheet problems in the learning experienced by WES students? 

• Do WES students develop their capacities to be resourceful problem solvers who use 
multiple approaches? 

• Do WES students become more self-reliant thinkers? 

Both student and instructor assessments of the role of worksheets indicate that the worksheets 
are critical to the effectiveness of the WES approach. 

The qualitative data indicate that a key outcome for WES students is an increased capacity to use 
multiple ways of problem solving. Quantitative data appear to support this conclusion. 

Qualitative data indicate that a key outcome for WES students is an increase in self-reliance and 
problem-solving abilities. 

2. Are the WES instructors (both student assistants and workshop instructors) interacting with 
the students in ways that facilitate student learning, and if so, what are the key features of 
these interactions? 

The student interviews indicate that the way the instructors and senior peers interact with them 
(“there for you when and only when you need them”) is crucial to the effectiveness of the WES 
approach. 

3. What roles do work groups play in this learning? What are the necessary characteristics of 
work groups that effectively foster learning? 

The student interviews indicate that students experience their work groups as the most important 
element contributing to their success as calculus students. 

4. Are course retention and performance outcomes for WES students comparable to the out- 
comes achieved by students in other workshop programs, when analyzed by sex and ethnicity? 
Do WES students persist longer in science, mathematics, and engineering (SME) majors than 
non-WES students? 

Course performance outcomes for WES students overall, as indicated by adjusted mean grades, 
are comparable to those achieved by students in other workshop programs. When analyzed by 
sex category, in Math 221, WES males statistically outperformed their non- WES peers but WES 
females did not. However, in Math 222, WES students of both sexes statistically outperformed 
their non-WES peers and WES achievement for females was especially impressive in terms of 
its absolute magnitude (adjusted mean grade of 3.47, or “AB”), in comparison to non-WES 
women (adjusted mean grade of about 2.54, or “BC”) and in comparison to WES males 
(adjusted mean grade of 3.15). Analyzed in terms of ethnic category, the numbers of students in 
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under-represented categories are so small that statistical tests could not be performed in a mean- 
ingful manner. However, there is a positive descriptive pattern of adjusted mean grade differ- 
ences within each ethnic category for both Math 221 and Math 222. Persistence data are not yet 
available due to the pilot nature of the outcomes. 

Overall, we conclude that WES students leam calculus better than discussion section students, largely 
because the WES approach better matches the way learning occurs than does the discussion section 
approach. The qualitative assessment findings suggest that learning is a complex, stochastic process 
requiring multiple and varied opportunities for understanding to develop. The WES environment 
provides continuous opportunities for complex, stochastic learning moments. By contrast, the 
discussion section process, which depends on instructors giving their version of a clear explanation, 
and timed tests of individual students’ capacity to perform provides what many students experience as 
“one-chance-only” learning opportunities. 

This one-chance-only approach is relatively more dependent on the skills and personality of a 
discussion section instructor. Students who have weak instructors and/or fail to understand a concept, 
and/or perform below their level of understanding during timed tests thus are inclined to feel that 
either they missed their chance or don t have what it takes to understand the concepts and perform 
well. 

These contrasting features of the two approaches are summarized in bullet form below. 

1. Discussion Section Approach 

• Offers lecture monologues, encourages solitary work focused on instructor 

• Students get largely unidimensional, unidirectional (instructor-to-student) explanations and 
demonstrations, which result in more of a one chance only environment that is less likely to be 
effective for a wide range of student types. This conclusion is supported by research on student 
learning styles. 

2. WES Workshop Approach 

• Offers lecture monologues, solitary work, but also encourages instructor-student dialogues, senior 
peer-student dialogues, and peer-peer dialogues; 

• Students get multidimensional interactions; the environment offers challenging yet forgiving, 
numerous and varied opportunities for learning and thus is more likely to be effective for a wide 
range of student types. 



We also conclude that the WES instructors must provide a knowable set of necessary and sufficient 
conditions, as follows: 

flexible workspace 
extended worktimes 
challenging worksheets 
functional groups 

supportive but not intrusive instructors and peer-like senior peers 
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We caution that, while the WES approach appears relatively robust over different types of students 
and instructors, the approach itself may not be robust if any of these necessary conditions is missing. 



B. Implementation Processes and Outcomes for Administrators: 

Summary Findings 

We list here summary findings for each of the research questions (presented in bolded italics) which 
shaped the data collection and analysis process focused on implementation processes and outcomes 
for administrators. For a summary of the implementation process involved in deve oping an infra- 
structure and administrative process, see “analytic generalizations” (Section III.B.2) above. With 
regard to the outcomes to date, we list the following: 



1. What progress, if any, has been made toward developing an infrastructure and an 
administrative process that is sustainable within the institutional reward structure. 

• The number of WES workshop sections offered in 1994-95 increased as indicated in Part IV. 

• A WES Program Coordinator has been identified and will begin work as of summer 1995. 

The Program Coordinator will train the new fall 1995 WES TAs at Madison rather than 
sending them to UT- Austin’ s Emerging Scholars training program. 

• Program evaluation continued throughout 1994-95, focusing on the WES-for-engineers 
workshops and supported by the College of Engineering’s TRP grant. This evaluation is 
being conducted by the LEAD Center, a new infrastructural unit whose creation was facili- 
tated, in part, by the evaluation work produced on the 1993-94 WES Program. Access to 
student database records, necessary to support faculty/staff efforts to develop longitudinal 
evaluation is being facilitated by the Division of Information Technology. 

• No changes to faculty/staff reward structure were reported. 

2. What are the prospects for institutionalization and scale-up of the Program? 

Prospects for institutionalization and scale-up are good, in large part due to the Department 
decision to support the Program on a pilot basis through 1999 (see section III.C.2). 

3. What are the results of the formative evaluation activity? 

Formative evaluation information was presented to the Math Department and used to inform 
their decision about continuation of the pilot program. In addition, formative evaluation infor- 
mation has been and continues to be disseminated to Math Department faculty and administra- 
tors and relevant campus-level administrators in the form of written reports and an audiocassette 
program. Faculty and administrators report that these materials are valuable to them as they plan 

and revise the Program. 

4. What are the total instructional and administrative (including any evaluation and dissemina- 
tion) costs of the WES Program? 

According to calculations produced by the Chair, Richard Brualdi, for the October 1994 Math 
Department meeting, “Each WES section requires a half-time TA who is responsible for 4 stu- 
dent-credit hours. The additional 2 student-credit hours come from Math 298. Thus a WES TA 
contributes the same number of student credit-hours as does a half-time TA with two discussion 
sections of Math 221/222/223. Thus there are no added TA costs if one takes into account the 
extra student-credit hours generated.” Brualdi also calculated that the following per-section 
costs: $1,280 for an undergraduate student assistant; $500 for TA training; $2,480 for supplies. 
At this rate, the additional cost of offering 1 1 WES sections in 1995-96 would be $27,280. As 
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the Department urged the Executive Committee to hire a half time program coordinator to man- 
age the Program, this person’s salary and fringe costs also should be included to obtain the total 
instructional and administrative cost. Evaluation and dissemination activity has been funded by 
external grants obtained by individuals outside the Department and matched by College or insti- 
tutional sources. 

5. Are the costs of the program justifiable in terms of program benefits? 

It is too early to answer this question with certainty if value of the program justifies the addi- 
tional cost. 
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V. Recommendations 



A. Pedagogical issues 

1 The senior peers are critical to the success of the WES approach and are a bargain if instruc- 
tional costs are evaluated in light of instructional value. Therefore, this element of the Program 
should be given high priority in any funding decisions. 

2. Instructors should: 

• Do whatever is necessary to keep the learning environment rich (learning is stochastic) by 
drawing students into multiple interactions in which they talk mathematics. 

• Continue to use challenging, complex, yet appropriate worksheet problems. To sustain the 
goal of developing student self-reliance, instructors should continue the practice of not 
providing answers to the problems. Because instructors are not always able to check the re- 
sults of each group s answers to each worksheet problem, learning might be improved 
slightly if students were provided answer sheets at the subsequent workshop. (It is possible, 
however, that the instructor time needed to develop answer sheets would be too great, given 
the marginal improvement that answer sheets might provide.) 

• Continue to foster work group energy and identity by, for example: 

• fostering student groups that work. Sustain groups that are dynamic and productive, 
and reorder groups that are only marginally functional. We suggest that groups be 
structured by the instructor for one session each week in order to avoid group stagna- 
tion and foster students abilities to interact with a range of different students. 

• misting the students, thereby challenging them to access and develop their own re- 
sources. 

• coaching students in how to take the course exams by occasionally giving students 
ungraded quizzes The quizzes should be timed or untimed depending on whether the 
lecturer uses timed or untimed exams. Comments on the quizzes should be focused on 
helping students develop more effective test-taking strategies (e.g., show all your 
work). 

B. Out-of-Class Issues 

The WES Coordinator should follow policies that foster group energy and identity: 

• The social events, and particularly the registration week party, are useful. 

• The WES dorm floor is useful. 
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C. Implementation Issues 

1. The variety of options for taking calculus should be expanded by advertising WES to all students 
and meeting the resulting student demand. (Instituting new options will require that the Depart- 
ment deal with new issues that will arise, such as changes in testing and grading procedures.) 

2. On an experimental basis, offer larger WES sections, adding more senior peers (who are inex- 
pensive and effective) in order to keep the cost/student ratio down. 

Rationale : It appears that the WES approach is robust enough to remain effective for larger 
section sizes. 

3. The Math Department should evaluate the student learning outcomes of the various teaching 
approaches offered in its calculus program in terms of both efficiency and effectiveness of in- 
structional dollars spent on each approach. This would entail including course drops in student 
database analyses. These data may be useful for informing department policies. 

4. The process for obtaining student database information mustbe simplified. 

5. The senior peers should be paid for time spent on WES office hours. 

6. Formative evaluation research should be continued. In addition, longitudinal data analysis on 
WES student cohorts should be conducted in order to track students into their majors and to 
graduation. 

Rationale: Formative evaluation information can shorten the learning curve and minimize 
difficulties during program scale-up. Longitudinal data can be used to help the Department and 
College develop and sustain maximally effective lower-division student learning environments 
that expend instructional dollars with maximal efficiency. Evaluation research data can help the 
Department and College develop and update dissemination materials on the WES Program. 
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LEAD Center Project on WES 
Interview Guide 

Fall 1993, Calculus 221 
Beginning of Semester Interview Sessions 



WES Students 

Introductions and presentation of Informed Consent form 

Briefly review points from the Informed Consent, focusing on how the evaluation and assessment 
process relies on learning about what in the course helps or does not help students understand how to 
do and to appreciate calculus. The evaluators are allies who can help students communicate things to 
the instructors that might be awkward for students to say directly. Check that the student is comfort- 
able with the tape recorder. If not, just take notes. 

Background 

* Please tell me a little about the schools you attended since about the sixth grade. Help me under- 
stand what led to your decision to come to UW-Madison for college, [teachers, family, friends, 
counselors, special experiences?] 

Expectations 

* Standing on the threshold of your college experience, what do you expect it will be like? What 
do you expect from your college years? 

* How do you plan to use your calculus class? Tell me a little about ideas you have about a major, 
or possible plans for your future professional life. 

* What are your expectations of your calculus course? In particular, what do you expect of the WES 
program? 

First Impressions 

* What are your first impressions of UW-Madison? 

* What are your first impressions of your calculus lecture ? 

* What are your first impressions of your WES workshop? 

WES Invitation 

I understand that you received a letter and brochure inviting you to join the Wisconsin Emerging 
Scholars Program [present copies]. 

* What went into your decision to accept that invitation? 

* Do you know any other students who got an invitation, and what did they do? 

* Can you remember anything about the letter and brochure that you particularly liked? What, if 
anything, would you change to make them better? 
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Party 

How did you like the party? Did you think the VIPs should have been introduced? 

Gender 

* Do you think that women’s and men’s experiences in math classes are different? 

* Do you have any observations about gender differences in the WES class? 

Additional 2 credits 

* Do you think the extra 2 credits for WES participants is a good idea? 

Closure 

* Is there anything else you think that someone trying to understand student reactions to this course 
should know at the beginning? 

* Do you have any questions for me? 
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Discussion Section (DS) Students 

Introductions and presentation of Informed Consent form 

Briefly review points from the Informed Consent, focusing on how the evaluation and assessment 
process relies on learning about what in the course helps or does not help students understand how to 
do and to appreciate calculus. The evaluators are allies who can help students communicate things to 
the instructors that might be awkward for students to say directly. Check that the student is comfort- 
able with the tape recorder. If not, just take notes. 

Background 

* Please tell me a little about the schools you attended since about the sixth grade. Help me under- 
stand what led to your decision to come to UW-Madison for college, [teachers, family, friends, 
counselors, special experiences?] 

Expectations 

* Standing on the threshold of your college experience, what do you expect it will be like? What 
do you expect from your college years? 

* How do you plan to use your calculus class? Tell me a little about ideas you have about a major, 
or possible plans for your future professional life. 

* What are your expectations of your calculus course? 

First Impressions 

* What are your first impressions of UW-Madison? 

* What are your first impressions of your calculus lecture ? 

* What are your first impressions of your calculus discussion section? 

Gender 

*■ Do you think that women’s and men’s experiences in math classes are different? 

Do you have any observations about gender differences in your calculus class? 

Closure 

* Is there anything else you think that someone trying to understand student reactions to this course 
should know at the beginning? 

* Do you have any questions for me? 
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LEAD Center Project on WES 
Interview Guide 

Fall 1993, Calculus 221 
Mid-Semester Interview Sessions 



WES and DS 
Background 

Has anyone in your immediate family attended college? 

Did any friends or associates go to college? 

When did you decide you wanted to go on to college and why? 

What about your major interests you? 

Was there any event or any person who influenced your interest in your career? 



Study Habits 

Describe your study habits outside of class. 

WES ONLY 

Are you in Fletcher [the dorm floor where many WES students live], and if so, does that affect your 
study habits? 

Do you ever study in the WES classroom? 

What do you think about the room? (probe this) 

WES and DS 
The Course 

What are your impressions of how the WES program/your discussion section is going? 

Are there any things you particularly like? 

Are there any things which could be changed? 

Does your instructor’s personality influence your experience in this course? 

WES ONLY 

What do you think about the undergraduate teaching interns? 

What do you think about the worksheets? 

WES and DS 

What did you think about the first exam? (probe this) 

Is there anything specific about being in the WES program or your discussion 
section that made a difference on the exam? 
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Classroom Dynamics 
DS ONLY 

Describe the dynamics of your discussion section, (friendly, competitive, obnoxious, 
cooperative) 

What role, if any, does competition play in your experience of learning math? 

WES ONLY 

Group work is a key part of the WES program. Describe the group dynamics that you 
have experienced, (friendly, competitive, obnoxious, cooperative) 

What role, if any, does competition play in your experience of learning math? 

In your group, do students work together to solve the problems or do they work on 
their own and then ask each other questions? 

Who decides who goes into the groups? (probe this) 

Which way do you prefer? Why? 

Would you prefer to stay with one group all semester? Why? 

WES and DS 
Overall 

How do you feel about calculus now? 
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LEAD Center Project on WES 
Interview Guide 

Fall 1993, Calculus 221 
End of Semester Interview Sessions 



WES Students 
Introductory Questions 

* Are you planning to take 222? 

* What will you remember about this course a year from now? 

General Questions 
EXAMS 

* What do you think of exams when 60% = C? 

* Tell me about the effect of 221 grading on you? 

* How are you planning to study for your final exam? 

EFFECTS 

* Do you think this course has made any difference in the way you think? If so, why? Give me an 

example, when, how. . 

* What relevance does this course have in the big picture of your college education as you see i 

* When you think about this class, what are two or three times that stand out for you? 

* What is your opinion of the WES social activities? List them and evaluate each. 

* What do students you know in regular discussion sections think about WES? 

* How has this course turned out relative to your expectations? What stands out as being different 

or exactly like what you thought it would be? 

* If you had it to do over, would you take WES or a regular discussion section? Why. 

DIVERSITY 

* The WES classes are about 50% men, but in most calculus classes, 70% of the students are men. 
What would the course be like if 70% of the class was women? 

* The ethnic diversity of the WES class is much greater than in typical UW-Madison classes. 

Would you say this has affected the way you are learning calculus? The way you feel about the 

class? 

FINAL QUESTION 

* Imagine you have a cousin who is planning to come to UW next year. How would you advise her 
or him as far as taking calculus? 
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Discussion Section Students 

Intro 

* Are you planning to take 222? 

* What will you remember about this course a year from now? 

General Questions 

EXAMS 

* What do you think of exams when 60% = C? 

* Tell me about the effect of 221 grading on you? 

* How are you planning to study for your final exam? 

EFFECTS 

* Do you think this course has made any difference in the way you think? If so, why? Give me an 
example, when, how. 

* What relevance does this course have in the big picture of your college education as you see it 
now? 

* What was the most helpful thing for you in learning calculus? 

* When you think about this class, what are two or three times that stand out for you? 

* Here’s a situation: (describe cooperative learning groups, harder problems, and 6 hours a week, 
and not much chance to ask questions about the homework, earn 2 extra credits). Is that some- 
thing you think you’d be interested in? [For students taking 222,] would you like to 222 with 
these methods? 

* How has this course turned out relative to your expectations? What stands out as being different 
or exactly like what you thought it would be? 

* If you could design the way this course is taught, what, if anything, would you do 
differently? 

GENDER 

* In most calculus classes, 70% of the students are men. What would the course be like if 70% of 
the class was women? 

FINAL QUESTIONS 

* Imagine you have a cousin who is planning to come to UW next year? How would you advise 
her or him as far as taking calculus? 
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Appendix B 



Sample Notes from Observations of a Large Lecture, 
a Discussion Section, and a WES Workshop 

OBSERVATION of LECTURE 1 

11/12/93 

Lecturer 1 arrives just before 9:50, wearing jeans and a plain black sweater. Several WES students 
are in the very front of the room. Noisy room until Lecturer 1, holding a microphone in left hand, 
begins writing on the board with his right hand, at which moment there is almost immediate quiet 
form the students. He tells that class that he will not talk about the exam they took last night because 
some of them will take the exam on Saturday (tomorrow). He has 9 blackboards, 3 of which are 
always hidden because they slide behind one another. 

He lectures. (I can’t see the several WES students in front of class] Seems to be having a problem 
and comments “I think I’d better take another sip of coffee” and smiles sweetly at students [some 
quiet laughter] 

9:00 Only one or two latecomers to the class. Students do not appear sleepy, and most are assidu- 
ously taking notes. His lecture style is very good: his writing on the board is very clear, and his voice 
is clear as he talks while he writes each mathematical symbol on the board, with many pauses ena- 
bling lots of time for taking notes. He is careful to turn around and address students whenever he is 
not actually writing. However, he does not appear to be catching students eyes for their responses, 
and does not pause to ask for questions. 

9:06 Still no pause for questions, but he admits to class that he forgot something in one notation. 

He compares two notation methods, and explains the respective strengths and weaknesses of each. 

9:07 Pauses, “OK, let’s do a few more..” [students stir, a few cough]. He goes right onto the next 
example, and now it appears he is keying his eyes on a set of students in the front right side of the 
audience. 

9:12 He sweeps his eyes over the whole audience, and shifts to another point. A girl near me turns 
to her neighbor as says, “what?” referring to what Lecturer 1 is doing now. He produces two solu- 
tions. Pauses and looks out to general audience, and says, “So what does this tell us?” 

[metacomment] then tells them he will do a third now because he suddenly remembers a trig formula, 
and derives that. Then says (while erasing the board), “So let me summarize that. I got three differ- 
ent answers, so what’s going on?” and writes all three answers one above the other. Tells them to 
“pay close attention to the following argument because there’s a mistake I’m making.” Illustrates 
how something must have gone wrong because sin squared x plus cosine squared x = 0. So what 
went wrong? He pauses and announces he will get more coffee, and while he does, one student (from 
right front) states briefly what went wrong (the constants weren’t the same). He says, yes, and goes 
on to explain. Then he tells them he will write down the correct argument, and does so. During this 
time, he checks out audience a little more, but tends not to look at left side of class, (maybe because 
he holds the mic in his left hand.) 

9:20 Students here and there interact a little with their neighbors. 

39 
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9:22 Having finished one point, he moves directly to the next, pausing to erase some blackboard. 

He uses the typical “math professor question,” the kind which the professor immediately answers 
himself. He gets a bit friendly, mentioning to them that he’s seen this integral before, and thus knows 
how to derive it. He tells them he just knows this particular approach. He makes lots of guiding 
metacomments like, “This looks more complicated, except if you look closely, it’s not more compli- 
cated,” or, “Now we’re in trouble — how do we get rid of this U?” 

Note: He moves quite comfortably, with large, confident steps and an assured, charming manner — I 
would say very “well socialized” as a math professor, has all the subtle mannerisms that convey 
authoritative understanding of this discipline. 

9:27 In response to one of his questions, he elicited “inverse tangent” as a response from several 
students in the front of the class. When he pauses to erase board, some students tend to look at each 
other and chat a bit. 

9:32 Goes directly onto another example, again without pausing to ask for questions. Most students 
appear to still be attentive overall, but a few have closed eyes or are slumped over. 

9:36 He finishes another problem and pauses to erase, increasing percentage of students starting to 
lose it, looking at their watches, or looking zoned out. Lecturer 1 metacomments on his pause that he 
“is trying to think up a complicated example.” One student in the front laughs, and Lecturer 1 re- 
sponds to this with a friendly smile. Some students perk up and take notes, but others continue 
dozing, one nearby pulls out a candy bar and munches. Lecturer 1 makes a cute comment that gets 
several giggles from front of room — some in back of room start to pay attention (“what did I miss? ). 

9:40 A couple students put on their coats. Lecturer 1 starts to speed up a little, aware that he is up 
against the bell. At the bell, students immediately throw back their writing boards (very noisy!) while 
he is talking, and get up to leave — pavlovian, I would say. Lecturer 1 finishes his phrase and stops. 
One female student approaches for a quick question, then another girl goes up and asks a quick 
question. A guy approaches. 
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OBSERVATION OF DISCUSSION SECTION 1 

11 / 11/93 

11:02 Instructor 1 shows up, and asks right away for a question. A woman asks a question. 

Instructor 1 at board asking questions and pausing long enough to get some responses from class. 

He pauses, asks questions of himself and gives them a moment to think over the question. 

1 1:09 A guy walks in, and Instructor 1 stops and says, “Jason, welcome, we’re trying to figure out 
problem that I don’t know how to do.” He asks if they remember a certain problem and when no one 
responds he makes weird sounds and falls on the floor, and hollers, “Is anyone awake here? 

Students laugh and he observes that doors are closing down the hall, “as happens so often when we do 
our calculus discussions.” Class doesn’t particularly liven up, however— one woman is checking out 
the timetable for semester 2, a few students taking notes. 

11:12 Instructor 1 : “Everyone keep this in mind, it’ s very important...” 

Note: Instructor l’s efforts to get response seem to illustrate the structural situation of “students as 
observers of teacher performing” to an extreme — both good and bad: He makes the best of this 
structure, and in doing so, still gets relatively low-level responses. He illustrates an inherent charac- 
teristic of the structure, which is a tendency to encourage student passivity. 

11:15 “I think we should pause and see if I didn’t screw that up, [chortles loudly] .” 

1 1:17 One guy volunteers that Instructor 1 should have factored something. I: “You’re a sharp 
shooter, aren’t you — it’s easy to see it now.” [metacommenting on his position as the performer at the 
board] 

11:19 A woman (Jill) sneezes, and Instructor 1 says “Bless you, Jill, maybe you don’t know but we 
have a no sneezing rule here. I run a tight ship!” 

1 1:20 Guy asks another questions — clarification on the last problem 

1 1:21 Instructor 1 asks for other questions, and a woman raises hand and asks for a certain number 
problem. 

“Do we have a formula for this one? not exactly, right, so what’s the idea here?” Writes out a for- 
mula and asks, “Is that clear?” (no response from students) and he explains (answers his own ques- 
tion). “What shall we call the long base? Let’s call it x, my favorite letter. What shall we call the 
short base? Y....” etc. Note: he leads them through how a person should think a problem like this 
through. “Now we’re starting to think I could have used a better letter than x.... writes up 
SOHCAHTOA. Walks them through the problem, holding a dialogue with himself, commenting to 
them..”Do you have an idea about what to call this, or is it just me? How about if I do this? Will 
this baffle everyone?” Derives something. 

Note: Students remain pretty quiet, passively watching. Two guys in front of me pass a comment, not 
about math, and woman to my left still works on her timetable for next semester. 

1 1 :25 “Is everyone with me up to there?” (no response from class except for some copying down 
what’s on board) 
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In response to Instructor 1 ’s next rhetorical question a couple guys near the front rumble an answer. 
(Guy to my right is looking out the window and sort of copying down what Instructor 1 puts on the 
board.) 

1 1 :34 One woman notes something to her neighbor. Students quiet and watching, some take notes. 
“Was that suspenseful, were you all on the edges of your seats?” Instructor 1 asks at the end of the 
problem. 

11:36 Woman reading the timetable asks Instructor 1 a general question about the process of using 
max-min problems. (She has no math papers or books out — only her timetable.) He asks if they can 
graph a problem and a couple guys point out that he always makes mistakes when he does this kind of 
graph. He laughs and claims he will conquer this one. Does the graph and notes, “I should take some 
more math before I start teaching.” 

1 1 :40 Jon asks a question about the same problem that he just described, and they interact twice a 
clarification question. Students continuing to pay attention and take notes but not much energy in the 
room. 

1 1 :44 Instructor 1 asks, “Perhaps now is the time...” He asks them to remember their love of 
calculus. Jokes with them about how asleep they look and reminds them of the test they have tonight, 
and says that they should remember they love what they’re doing, especially just before a test. He 
leads them through the “Yeah calculus” chant once, and then again in an almost whisper. 

1 1 :45 Instructor 1 asks for one more question in the last 5 minutes, and a woman asks for a related- 
rates problem. A woman sneezes, and he comments that she’s been here since the first day and still 
doesn’t know the rules either. They do the problem, a couple comments from middle of room in 
response to his questions. 

1 1:48 Same woman as asked first question asks for a clarification, and a guy (one of the more active 
ones) raises his hand too for a clarification. Instructor 1 comments again on how his mind operates. 
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OBSERVATION OF WES WORKSHOP 1 
11/11/93 

(All names have been replaced by psuedonyms. Work groups are identified by their tables, as in T1 
for Table 1.) 

I arrive about 5 minutes late. 

Student Assistant (SA) B present as student intern, and a large bowl of Halloween candy (presumably 
Instructor 1 brought it). Students go over and get candy from time to time. 

Same students as on 1 1/9, except Carla not there and same configuration except table with 4 women 
is broken into 2 groups of 2, with Sheila and Ellen at one table, and Linda and Kathy at other one, 
and Todd is at the table with Mark, Charles and Peter. I chat with Sheila and Ellen, who explain that 
they two usually work together on the algebra stuff at their table of 4 anyway— comparing answers 
with the other two, so that’s why they decided to sit separately today. They tune in to Victor’s voice 

at next table. 

Groups immediately start working on their problems — I was about 5 minutes late so missed the 
introductory remarks Instructor 1 might have made. SA-B at T1 spent some time talking about how 
not to get hung up with test anxiety. 

1:50 Instructor 1 not circulating, but on the side, working on a problem. Ray and Victor, sitting 
next to each other, somewhat belligerent as usual, with Diane clearly holding her own in interactions 
with Victor. Randall works by himself as before. 

1:50 SA-B at T 1 for quite a long time. 

1:54 Kathy goes over to Instructor 1 to ask a question. Instructor 1 spends some time at T2, talking 
with Mark about how to think about a general concept, while Charles and Peter and Todd all work 
quite independently until Todd asks Charles a clarification question, and Instructor 1 keeps talking 
with Mark. All the while, SA-B working exclusively with Debbie at Tl. 

2:05 Instructor 1 to T3. Diane rolls over to check out what Kathy and Linda are doing 
atT4. 

2:10 SA-B to T2 to work with Mark, and Instructor 1 off to the side again, apparently working on a 
problem. Then Peter calls her to T2 for a short question. 

Meanwhile, at T5, Victor observes that the women have to work with the women (as Diane is over at 
T4), and I co mm ent that Rodney is an exception, and that he (Victor) is into making sociological 
observations. He asks if I’m a sociology student and I say I’m an anthropologist. He asks, “a doc- 
tor?” I say, “yeah,” and he goes “whoa” (I think he is ambivalent about my being here — likes but 

doesn’t like the attention). 

2:13 Instructor 1 to T4 

(Note: Students call her “Dr. X.” There are no calculators in the room.) 
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While at T4, Instructor 1 helps Kathy and Linda by asking leading questions, giving hints but no 
answers. The women seem to like it as they struggle with the problem. While Instructor 1 is working 
with them, Ray and Diane come over to see what Kathy and Linda got, and then Victor comes over 
too. Instructor 1 eventually leaves T4, encouraging Kathy and Linda to go for the big picture 
(remember what the problem is asking and try various values), and goes over to T5 where all pay 
attention. 

2:20 SA-B to T3, and the rest of the room doing the usual — Todd by himself and the other 3 at T2 
very engaged, Debbie by herself at T1 and the other 2 working together. Ray and Victor break into 
mild and friendly pushing, and Victor keeps catching my eye. 

2:27 Instructor 1 also goes to T3, along with SA-B. All 4 at T5 interacting with Linda and Kathy 
about a problem. 

Until 2:40, Instructor 1 spends quite a while with T2, where they describe how they set up the 
problem and want to get from her some idea of whether they are thinking it through right. She gives 
them hints, and they finally decide they have it right, but they don’t want to bother actually doing the 
calculations. 

Meanwhile, SA-B is at T5, working a lot with Diane, and Dennis from T1 wandered over at one point 
to ask Instructor 1 a question. He returns and keeps working with Carrie, while Debbie continues to 
look lost. Until 2:52, 1 stayed at Tl, listening to Instructor 1 work with Dennis, Carrie, and Debbie. 
Debbie follows along, even though she does not speak much. Instructor 1 works very carefully at 
helping students with problem 4, articulates relationships between the general rules they are learning 
and the particular problems; she notes when they get lost in the details of a problem and encourages 
them to think through, always careful not to give answers, and to pull out connections. 

At T4, Kathy and Linda work together to work out problem 4, each one constantly interacting with 
the other to sort out what they need to do to set up the problem and find the answer — it seems that 
each is contributing, and that they get through it faster because of talking. Meanwhile, next to them, 
SA-B works out the algebra of another problem they did that SA-B believes is wrong. 

Over at T5, Victor and Ray are finishing up problem 5 and moving onto problem 6. These two really 
enjoy competing with each other, but it is in a very friendly way. It seems that Rodney is not in the 
same place as they are, still working on an earlier problem. 

At 3:15, as students are starting to leave, SA-B announces that she is going right home and can be 
reached there by phone between now and the exam time tonight and kids can call her there. 

Dennis, talking with Instructor 1, tells Instructor 1 not to worry about them tonight, and Instructor 1 
says, “I’m not worried about you!” and Dennis and Carol indicate that they are a bit worried about 
themselves. Meanwhile, Sheila leaves and Ellen grabs her sleeve and says Ellen might stop by. Over 
at T5, Linda and Kathy have come over, as has Ellen and Instructor 1, to compare answers to ques- 
tions that people got to different problems, and Kathy is real happy that an answer she and Linda got 
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is just what others got— Kathy and Linda slap hands, Kathy looking shyly about this sort of “guy” 
thing to do Ray and Victor still working with Instructor 1 past the class time, intent on finishing 
every problem on the worksheet. Todd and Mark still working at T2, Rodney leaves and Ellen says, 
“Bye, Rodney.” Mark rolls over to Rodney and Victor to compare an answer as well, while Linda 
and Kathy go back to their table with SA-B and keep working on a problem. Ellen is hanging out by 
Ray and Victor, and Diane, and Ray call details of how to solve a problem over to Linda, Kathy, and 
now Diane, who has moved over to T4. Victor speaks out the problem solving process he’s working 
on and whenever he makes an error, Ray goes “ENHGH like a buzzer on a talk show. 

3:25 Tom calls Instructor 1 over with a question, and Diane is working still with Ray, and SA-B 
still with Kathy and Linda. Todd by himself, still working too. I have to leave, so am unable to see 
how long this last bunch stays on. 
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Appendix C 



The UW-Madison Learning Through Evaluation, 
Adaptation and Dissemination (LEAD) Center 

In August 1994, the LEAD Center opened its doors on the UW-Madison campus. The first of its kind 
in the nation, this Center supports UW-Madison faculty engaged in educational reform activities at 
both the baccalaureate and graduate levels. 

The LEAD Center’s mission is implicit in its name. 

L Student learning is our focus. 

LEAD clients are faculty and staff who seek empirical answers to the question, “How are my 
students learning?” We help our clients answer this question. 

E Formative evaluation is our approach. 

Evaluation is “formative” when applied researchers give faculty reformers feedback 

• while a program/activity is being planned, piloted, and scaled up, and 

• with the intention of improving the program/activity while it is under development. 

A Understanding of adaptation patterns is a primary goal. Our knowledge of how educational 
reforms have been adapted successfully at UW-Madison and elsewhere can help faculty clients 
establish their innovations. 

D Dissemination strategies are important products. 

Using evaluation information, we develop dissemination materials for faculty seeking to adapt 
the approaches of LEAD clients to their own educational settings. 

Additional LEAD Center services include the: 

• LEAD Colloquium Series . 

This series is designed to encourage campus-wide discussion of educational reform activity at the 

UW-Madison and nationally. 

• LEAD Library 

Our clients and others are welcome to consult our collection of materials on evaluation methods 
and educational reform processes and outcomes at UW-Madison and nationally. 

• LEAD Colleagues Consulting Group (CCG) 

We arrange planning meetings between LEAD clients and UW-Madison experts in evaluation and 
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[I. Opening Quotes, Introduction, and Description of Program] 

***[music]*** 

...the structure of the WES Program is we're in individual groups, in groups of three, 
maybe four people, and our senior peer, our senior TA, usually a professor, just kind of 
moves around the room kind of making sure that you are on task. But she doesn't hover 
you and make sure. She doesn't over you and make sure that you do it, but she does kind 
of keep an eye on you. That, you know, gets you to spur on and continue with your 
work.. (African- American male WES student) 

*** [music]*** 

I like smaller classes because you really get to know people. It's not intimidating to go in 
there and you never know anybody and it's a huge lecture hall. And that's what's good 
about having the discussion sections, is that you really get to know people—but actually 
last semester in my discussion section I think I knew one person~no one really interacted. 
This is really, really different, the WES thing is. So I like smaller classes where you get 
to know people, and you can talk about stuff and it's not just get in there, sit down, take 
notes and leave--that's an easy way to get tested, I think. I test well and stuff like that, 
but I don't really get much out of the class that way... (White female WES student) 

***[music]*** 

..The human aspect I think is the most important thing. The human aspect in my opinion 
allows you to really understand the material because I find that with most of the courses 
there's this student-teacher, and it's so formal that it's almost cut and dry trying to learn 
something. But when you kind of break down those barriers and, you know, you have a 
student-to-student, a friend-to-ffiend, and learning the material becomes much easier. It 
doesn't seem so cut and dry. I mean, while you can talk about non-math things, you can 
talk about math things. And I find that is the most productive way to learn something. It 
may sound counter-productive, but for me it's actually very productive. (African-American 
male WES student) 



*** [music]*** 

Susan: Welcome to "Evaluation of the Pilot Wisconsin Emerging Scholars Program, " an 
audio presentation featuring the calculus program which the students you just heard were 
describing. My name is Susan Millar, and I am hosting this program with two of my 
research colleagues, Baine Alexander, an anthropologist ... 



Baine: Hi. 



Susan: and Heather Lewis, a graduate student in the University of Wisconsin-Madison Math 
Department. 

Heather: Hi. 

Susan: Before we get started, we'd like to briefly explain how the audio program is 
structured. It begins with an introduction and description of the Wisconsin Emerging 
Scholars - W E S, or as we like to call it, the "WES" program. Next, we provide a students' 
perspective of some of the problems the WES program addresses by presenting excerpts from 
interviews with students who are in the traditional calculus program. Following this 
"problem statement" is the longest portion of the program, which features an analysis of the 
WES Program based on qualitative research interviews and observations. This section 
includes many excerpts from interviews with WES students and instructors. 

Baine: After our presentation of the key qualitative factors which help explain how the WES 
program works, we feature a quantitative analysis of the program. We hear from Professor 
Joel Levin, of the University of Wisconsin-Madison Department of Educational Psychology. 
Joel is an expert in statistical design and analysis in education research. The final portion of 
the program focuses on various problems which students and faculty encountered while 
implementing the WES Program. Listeners who would like more detailed information about 
the pilot WES Program can find information about how to obtain a written report by 
consulting the audiotape cassette case. 



*** [music]*** 



Susan: Baine, Heather and I acquired an in-depth knowledge of the Wisconsin Emerging 
Scholars program by conducting an evaluation. The evaluation was sponsored by the 
National Science Foundation's Division of Research, Evaluation and Dissemination, the 
University of Maryland University College's Institute for Research on Adults in Higher 
Education, and UW-Madison College of Letters and Science. 

Baine: The Wisconsin Emerging Scholars, or "WES, " Program, is a workshop-based calculus 
program that the University of Wisconsin-Madison offered for the first time in the fall of 
1993. The program is modeled on the "emerging scholars" program that Dr. Uri Treisman 
pioneered at the University of Califomia-Berkeley in the early 1980s. Treisman observed that 
the African-American students who enrolled in calculus at Berkeley had incoming placement 
scores that were comparable with other students', but that their calculus grades were lower 
than predicted and they dropped out at higher rates. He also observed that Asian-American 
students tended to do better in calculus than students from other ethnic groups. 

Susan: To understand what might be causing these differences, Treisman conducted an in- 
depth observational study of calculus students. What he found was that the Asian-American 
students worked collaboratively, and the African-American students worked separately. 
Treisman's findings led him to design an "honors program" for under represented ethnic 
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minority students— a "workshop " approach based on collaborative work. He found that the 
students in this "emerging scholars" program did significantly better than comparable 
students who studied calculus the traditional way. 

Baine: Variations of this workshop program were then adapted at other colleges and 
universities, often with comparable success. Among these additional "emerging scholars" 
sites is another program which Efraim Armedariz and Uri Treisman instituted at the 
University of Texas at Austin Dana Center for Mathematics and Science Education. Most of 
these programs were established at institutions that have relatively high proportions of ethnic 
minority students. And none, except for the programs which Treisman himself started, were 
established at major research universities. That is, until the University of Wisconsin began 
its program in 1993. 

Susan: Members of the UW-Madison Math Department were acutely aware that calculus 
persists, both nationally and at Madison, as a barrier preventing many students— and 
particularly under represented minorities and women— from entering mathematics-based 
disciplines. So with Treisman's encouragement and technical support from the UT-Austin 
Dana Center, a Madison Math Department committee instituted the WES Program, on a pilot 
basis. 

Baine: Along with Treisman, this WES committee suspected that the traditional discussion 
section was not providing the kind of learning experiences that many students need. Thus, 
they sought to replace the discussion section with the WES workshop format, believing that it 
would work more effectively. 



*** [music]*** 

Susan: I think we should pause here and ask Heather to describe how the Wisconsin Math 
Department's calculus program is organized. 

Heather: At UW-Madison students taking the three-semester calculus sequence— Math 221, 
222, and 223-traditionally enroll in a lecture which meets for 50 minutes three times a week 
and in a discussion section which meets twice a week for 50 minutes. Most of the lectures 
enroll over 300 students each and are taught by a faculty member. The discussion sections 
average 20 students each, and each is taught by a graduate teaching assistant, or "TA. " The 
TA reviews the lecture and works problems on a blackboard in response to student requests. 

Susan: Heather, how is the course graded? 

Heather: All students receive grades based predominantly on their performance on midterms 
and finals given in the large lecture. Only a small portion is based on homework and 
discussion section participation. For each large-lecture midterm or final, all the exams are 
graded during a single session by the entire group of TAs, with one TA grading all 300 
students on the same problem. This process ensures that each TA has only a small part in 
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grading his or her own students' exams. 

Baine: So how is the WES Program different? 

Heather: Students in the WES program attend the large lecture, do regular homework 
problems, and are graded in the same fashion as everyone else. But instead of enrolling in a 
discussion section, these students enroll in a "workshop. " Participation in a workshop allows 
them to receive two "special topics" credits (graded pass/fail) in addition to the standard five 
calculus credits. A workshop meets for two hours three times a week in a room reserved for 
the WES Program. In addition, WES students are encouraged to meet informally in this room 
whenever they wish, are invited to a few social events, and have the option of residing on the 
same dorm floor. 

Baine: In each workshop session, the instructors give the WES students "worksheets" 
comprised of problems that are more difficult and more carefully designed than the homework 
assignments. 

Heather: Sitting around tables in groups of three or four, the WES students work the 
problems together, while the TA and an undergraduate student assistant roam the room 
asking strategic questions and offering hints when particular groups are obviously frustrated. 
The workshop instructors avoid directly answering students' questions, and try to help 
students answer their questions themselves. 

Susan: In the pilot year, that is, 1993-94, the WES workshops averaged 17 students, and 
were instructed by a faculty member acting in the TA role and by an undergraduate "student 
assistant." The two faculty were both members of the WES committee. One was Mike 
Bleicher, who emerged as the Director of the Pilot WES program, and the other was Melinda 
Certain. 

Heather: The undergraduate student assistants were Denise Harbart and Dave Kung. 
Listeners should know that in some interview excerpts in this program, WES students 
sometimes refer to these undergraduate student assistants as "TAs" which is what graduate 
teaching assistants are called at Wisconsin. 

***[music]*** 

[II. Analysis of Traditional Discussion Sections] 

Susan: Our many research interviews with small groups of students and with individual 
students strongly confirm the WES committee's belief that this approach would result in better 
outcomes in calculus for under represented students— and for that matter , all students. 

Baine: To understand what "better" outcomes are, we need to understand student experiences 
in the regular discussion sections. Let's turn to our interviews with students and let them 
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speak for themselves about the problems the WES Program is trying to address. It's very 
important to note that the excerpts that follow are representative of interviews we conducted 
with regular discussion group students. The following three excerpts are from individual 
interviews with students in regular discussion groups. 

*** [music]*** 

I: How would you say the real learning takes place in your discussion section? 

R: I think the real learning just takes place when you sit down and do problems yourself, 
because you could sit and watch someone do problems all day but unless you actually sit 
down and do it yourself, you're not understanding how to do it, or you really, you just 
need the repetition of doing it yourself and understanding the methods and just sitting 
down and working them out. (White female WES student) 

I: ... if you talked about the real learning there,. ..just within the discussion section what 
would you say? ... it sounds like you're saying that doesn't happen in the discussion 
section. 

R: Probably when you just bring in a question on an assignment and then the TA will 
work through it. Otherwise, I mean, our TA does examples and stuff, but sometimes 
they're just a little, like over our heads, so I mean, we don't really, we just sit and watch, 
but we're all just like okay, well just let him do it... I mean, one of the only reasons I go 
is if I have questions, well, I always go to discussion, but the only time I really get 
anything out of it is if, it's like right before an exam and we go through the material or if 
I have specific questions. Otherwise, it just seems like we just kind of sit there and fill 
time just watching him go through stuff. He's really above us. He's so smart and 
understands it so well and sometimes I think he forgets that we don't, and hell explain it 
really just above our heads, and so we sit there and watch, but we really don't know what 
he's doing. So I guess that's really why I don't really like discussion, I think. 

***[music)*** 

I: ... how would you say the real learning takes place in your regular discussion groups, 
likp. maybe in your chemistry or your 221 regular discussion group. Is there a contrast 
there that you can draw for me? 

R: Yes, there is, big, big contrast. The "learning place" in a regular discussion would 
take place in lecture and at home when you do your homework (Hispanic male WES 
student) 

I: But not during discussion sections? 

R: No. During discussion sections we would only go to the discussion and ask for 



problems, that's 90% of the time. 

I: It's not a learning time? 

R: No, no. Last semester I had this TA, she sometimes wanted to teach, sometimes 
wanted to reinforce what the lecture did. But most of the time she would only answer 
questions about the homework and show the students how to do it, which for me wasn't 
any help. Since I really didn't learn in discussion but how to do the procedure. And then 
when I got to discussions I had problems with the, going back to the lecture, every lecture 
he would write down on the board the homework, and he would give you a series of 
problems and the difficulty would increase according to, you know, higher numbers. So 
in discussion we would only solve problems actually. It wasn't like a reinforcing of the 
discussion, we wouldn't talk about what the professor lectured about, it was just the 
problem solving. 

I: Right, doing homework problems? 

R: Yeah, doing homework problems that was it. And I would get in there, get there and 
with problems, but he like number 1 through 4 I had questions, I didn't get to the 60- 
something of the homework, and some students apparently took the course before, you 
know, or ... yeah, they had questions in the 60-, 40-something problems and I, of course, I 
didn't want to ask the teacher, "Oh, I have problems with number 1," because everybody 
would look at you. You feel kind of uncomfortable there. 

***[music]*** 

I: What kinds of things do you find particularly frustrating in a class? 

R: When I go and I come back, if I walk out of the class and I just feel like I didn't learn 
anything new, that was worthless, and it was a total waste of my time. I would have 
learned more staying home and reading the book than going to listen to someone talk. 
(White female WES student) 

I: What kinds of classes does that happen in? 

R: It tends to happen more in discussion sections. Just because a lot of times discussion 
sections are a review of what I've already learned in lecture. 

I: How would you say the real learning takes place in your discussion section this 
semester? 

R: It doesn't. In my discussion section this semester, we go over, he asks if anyone has 
any questions, we go over a couple problems from the assignment and that's it. I think if 
I didn't do the problems before hand, I would have no idea what this guy was talking 
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about and so in that case I don't think I really learn from discussion. I learn more from 
doing the problems before hand... I see a lot of variability within the discussion sections, I 
don't know if that's a disadvantage or not. Just that you can kind of get lucky either 
getting in a good discussion or a bad discussion. Like if your TA is really outgoing, does 
a lot of extra work for you, has review sessions for you, that can be really good, or you 
know, it just seems kind of like either you're lucky or you're not lucky, depending on 
what class you get into. It's hard to visualize anything different I guess. 

***[music]*** 

Baine: Susan, it's important to emphasize that these excerpts are representative of what we 
heard in our interviews with students in traditional discussion sections; most students 
conveyed that their discussion section was not a place where they really grapple with 
learning calculus. Rather, they experienced it as a place where points made in the large 
lecture were reiterated, and where they watched a math graduate student do homework 
problems for them. 

Susan: Many students also depicted their discussion section as a competitive environment, as 
a place where they often felt intimidated, afraid that their peers would think their questions 
were stupid. 

Baine: We also frequently heard students say that the value of a discussion section depended 
heavily on the skills and personality of the teaching assistant and was therefore a matter of 
luck 

[in. Brief Summary of Outcomes of WES Pilot Program] 

Susan: Let's turn now to the WES alternative to the discussion section approach. One feature 
of this approach is that it's both academic and social in orientation. Another feature is that 
the program adopts Treisman's idea that it should be an honors program for under 
represented first-year students. Incidentally, the first cohort was composed of some 35 
students, most of whom came straight from high school. Of these, half were women, and 
about 40 percent were from under represented ethnic minority groups. 

Baine: The Program organizers achieved this distribution of students by inviting all the in- 
coming minority students who had high pre-college math achievement scores and inviting 
only a small proportion of the white students with comparable scores. On average, the WES 
students had math placement scores and high school ranks comparable to the other 2000 
students in first-semester calculus. 

Susan: Yet, Professor Joel Levin, the statistician who analyzed the quantitative data about 
this program, found that the WES students' placement scores and their high school rank-in- 
class predicted that they were no more likely than the other students to do well. However, 
they completed the first two semesters of calculus with grade point averages that were half a 
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grade-point higher than predicted. Also, a much higher proportion of the WES students 
completed the calculus sequence. 

Baine: About 18% of the traditional, or non-WES, students dropped or failed the first 
semester and another 1 7% dropped or failed the second semester. By contrast, in the WES 
program, about 12% dropped the first semester and none dropped the second semester. No 
WES students failed either semester. We'll look at these quantitative results in depth later in 
the program when Joel Levin describes his statistical analysis. 

Susan: Our qualitative research interviews indicate that, compared with students in the 
traditional discussion sections, the WES students showed higher levels of confidence in their 
mathematical ability and greater comfort in performing calculus problems than the students 
whom we interviewed from discussion sections. 

Baine: What we learned overall, from both quantitative and qualitative measures, is that all 
types of students-women, ethnic minorities, and white males— performed better in the WES 
experimental workshop than their counterparts in the discussion sections. 

*** [music]*** 



[IV. Qualitative Analysis of WES/TD 

A. Three Learning Process Factors that are Critical and Interactive] 

Susan: In our view the outcomes with respect to grades are only a part of the story. Our 
interviews with students, combined with classroom observations, helped us develop some 
understanding of why the grade-point averages and course retention data for WES students 
are higher. Furthermore, our research indicates that a number of other things that are not 
reflected in the grade points are occurring for these students. As such, we use an expanded 
notion of course outcomes for students in the WES program. 

Baine: In this section of the audiotape program we describe three key learning "process” 
factors-that is, key elements that effect the learning experience-which consistently emerged 
during our interviews with the WES students. We conclude this section by delineating certain 
outcomes associated with these key learning process factors. 

Susan: These three key elements are: the group work, the worksheets, and the instructors. 

We find it useful to visualize these elements as three overlapping circles, because the 
interaction among them is essential to the functioning of the WES program. 

Baine: Rather than presenting these learning process elements and the outcomes associated 
with them in the terms that researchers use, we think it will be more interesting for listeners 
to hear the students and the instructors themselves present the kinds of remarks on which we 
are basing our analysis. We selected the quotes you will soon hear because they succinctly 
make some of the analytic points that emerged from our evaluative study. 
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[1. Group Work 

a. Group Work Makes Learning Easier and Fun] 

Baine: As you listen to these people, imagine their comments as giving shape to the first of 
our three overlapping circles. 

Susan: We will also ask you to visualize the "group work" circle as comprised of three 
interactive factors. One of these parts is the idea that group work cultivates a cooperative 
approach which makes learning easier and fun. Another is that group work provides support. 
And the third is that group work helps students develop both INTERdependence and a self- 
reliant INdependence. 

Baine: We turn now to excerpts from interviews in which students describe how the group 
work in WES cultivates a cooperative approach which makes learning easier and fun. The 
first excerpt is from a focus group interview and the second is from an individual interview. 

*** [music] ** * 

I: For you personally, what do you see as the key to WES? 

Rl: People, social, talking to everyone. Not like socially, but I mean, social calculus, if 
you can call it that. It's just talking about it and helping each other on problems and I 
don't know. (White female WES student) 

R2: I think it's the relaxed atmosphere, I mean, it's well you don't have to be all stressed 
out about learning it, I mean, you just go in there and everybody picks it up at their own 
pace and usually the paces conform after a while. (White male WES student) 

Rl: And I think it's kind of, like in regular discussions, sometimes I feel intimidated to 
ask questions because everybody else is quiet and they're listening to you, but like in 
WES we're just, "Professor Bleicher, get over here!" You don't feel stupid when you 
don't know something. 

R3: Yeah, and it's good too because you sit around with a bunch of people that after a 
while you really get to know and you just sit there and if you're having while you're 
trying to learn something, it make s it a lot easier to learn, if you're not having fun while 
you're trying to learn something, you usually don't learn it very well, so I t h i nk that's a 
big part of it too. (White female WES student) 

Rl: And I think it helps because we talk about our outside lives so we, like, have things 
in common and we can talk about like what happened yesterday afternoon or something 
like that, and I think it's just nice to have people to talk to. 

R2: Kind of diverts you from calculus if you're having problems or like if somebody can't 
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get there to answer your question right away you can just kind of like forget about it for a 
couple minu tes and then a lot of times when you go back to it, it's easier to look at. 

*** [music]*** 

I: ...this idea, that the process is what is most important for you with respect to this real 
learning question-can you relate that to WES, to the workshop method versus, let's say, 
the more traditional methods? 

Rl: well in the WES program, it's a little more complicated because you have the process 
of the mathematics itself and you have the process of being familiar with people and then 
that familiari ty will help you understand mathematics. (African-American male WES 
student) 

R2: The familiarity with your... 

Rl: With the peers, that allows me to understand math better, and I found that in my 
other classes I am ^attempting to do the same without even t hink i n g about it. I try to be 
familiarized with my peers, who I don't know as well, but I try, you know, "Why don't we 
get together and study this," you know, a more social than individual studying this. I tend 
to do that in my other classes, but it's difficult because I don't generally think about it so 
it's not a very strong impulse, but it is an impulse nonetheless. 

R2: In other words, you know that you want to do this, but you don't spend the time up 
and, 

Rl: Generally, yeah. It always kind of crosses my mind, well this would be great if I set 
this time up with this person and we get together and we start talking about and 
understanding and do the material. Because I find, perhaps a reason why I don't have the 
greatest productivity is that when I sit down and try to study something by myself, it s 
almost like a wall. I'm fighting myself trying to, I want to do this, and it generally 
doesn't go very far... 

R2: You get frustrated. 

Rl: I get very frustrated. 

R2: And when you're working with other people? 

Rl: When I'm working with other people, it's much easier, ... it's easier to relax, it's easier 
to, I mean, you don't have to think about every single step of the problem. Someone else 
could thinV of a process and then you could add on to it, or you could begin a solution 
and they finish it. It's not that you don't finish the entire problem, you do finish it, but 
they may add a little bit more you hadn't thought about before. 
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R2: So it just comes out of the interaction. 
Rl: Hm hmm. 



***[music]*** 

Baine: This last student described something which many of the WES students we interviewed 
experienced: compared to the more individualistic approach they are accustomed to, working 
cooperatively not only helps them learn the material, but also makes the learning process 
easier— and more fun. 



***[music]*** 

[b. Group Work Provides Support] 

Susan: In our second set of interview excerpts on group work, the student speakers also make 
contrasts between the individualistic approach and the cooperative WES approach. But 
instead of focusing on the idea that group work facilitates learning by making it fun, the 
students in these next excerpts highlight a second factor-that group work is a source of 
support. 



I: tell me about some time when you felt especially pleased with your work? 

R: After review sessions. Like the first chapter that we did this semester, I was just like 
222 is like the hardest thing I've ever taken in my life, I can't believe I'm taking this 
again. And that chapter 7 was just miserable, I didn't get it at all, I was in Professor 
Bleicher's office all the tim e and just was struggling. And then Dave gave us a review 
session and like the end of the first night and half way through the second night I'm like, 
"Yeah, I got it, I think." And then by the end of the second night I'm like, "Yeah, I got 
it, you know, I might not ace it but I'll do alright." And there's no way I would have 
done that without Anna sitting next to me and showing me how to do this integral and 
Tony showing me his cheat sheets to show me how to do that problem and Dave being 
willing to be there for four hours two nights in a row, so that's definitely a great thing. 
(White female WES student) 



***[music]*** 

R: One thing the WES program has helped with I believe is, even though you may be 
doing poor on one quiz or poor on one midterm, the social environment will encourage 
you to go on. Whereas if you were by yourself, I believe in general if people do badly 
they'll kind of almost toss it aside and go. "Well, I didn't do well, I'm not going to do 
well again, so just forget - I'm going to drop the course." The social environment is 
beneficial in that case, in that it will, you know, you get moral support from everybody 
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else. (African-American male WES student) 

I: Now does that come from the instructors as well or are you mostly talking about 
R: It comes from the instructors as well, yes. 

I: From all of them? 

R: Hm hmm The instructors will show concern if you are not doing everything to their 
expectation. They won't scold you for it, but they will show their own concern, which is 
very nice. 

I: And your colleagues, or the other students, what do they do? 

R: Well it really depends on who they are, but generally they're very comforting with 
those things. TTiey will try to help you through it. I know especially that people in my 
own group, we will tend to get together outside of class to actually study together. 

I: You mean like one person maybe did particularly poorly on a particular exam? 

R: Hm hmm, and outside the discussion class we'll actually get together and study 
calculus even more. 

I: You mean to help that person get through whatever it was that's causing the problem. 
R: Exactly. 



***[music]*** 

Baine: The support provided by the WES groups, particularly given that many of the students 
are in their first year at a large university, is essential with regard to retention issues. As the 
student we just heard explained, after receiving a low grade on an exam many students will 
drop the course, whereas, in WES the group support provides encouragement and additional 
assistance for students to continue. 

[c. Group Work Creates Self-reliance and Independence] 

Susan: All of the excerpts from students that we have heard thus far emphasize the central 
role that group work plays in their learning of calculus. We turn now to the third factor 
associated with why " group work" "works" for these students— which is that it fosters both 
INTERdependence and a self-reliant INdependence. These next interview excerpts depict the 
apparently contradictory experience which all the WES students we interviewed described: 
that group work develops their capacity for interdependence, but also develops their self- 
confidence and independence. 
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Baine: We present three excerpts to convey this point. The first is from, an interview with the 
two WES "student assistants, " both of whom were undergraduate math majors. In this 
segment, the student assistants generate their own analysis of the process by which group 
work fosters self-reliance. Following this segment, we present excerpts from two different 
focus-group interviews with WES students. In these excerpts, students make their own 
analysis of how and why the group work process facilitates their learning. 

***[music]*** 

I: So what's the benefit to the really good student, why would they want to do this? 

Dave: Because as I think both of us have found out this year you don't really leam a 
subject until you teach it. 

I: So you're speaking for yourself? 

Dave: Sure. 

Denise: Yeah, I agree. 

Dave: And man y students have commented on that. That they don't really understand the 
problem until they can explain it to somebody which again is why the best technique 
would be to, you know, to work at a table until one person understands it and then have 
that person explain it. It benefits everybody more. 

I: I see, so that in a nutshell, even though I didn't specifically ask you at this time, is your 
key to, your approach is to 

Dave: To have the students leam it themselves. 

I: Right. 

Dave: As soon as you know that they can leam it, to let them leam it. 

Denise: There are about three or four groups between both classes that are like that, that 
they work so well together and they know each other really well and they feel 
comfortable with that, where as soon as one person gets it you can just leave because you 
know they get excited and sometimes they start talking over you anyway, so, you just go, 
"Okay, you don't need me anymore," and then just move on to the next table. 

***[music]*** 
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I: How would you say the real learning takes place in WES? 

Rl: I'd say maybe like when you're trying to explain it to people that don't understand, 
because if you can explain it to someone else, then that means that you really understand 
it. And even if you didn't totally understand it when you go in there, if you explain 
something to someone, even if it's just a little part of something really big, you know that 
you understand that part really well, and if somebody else explains another part to you, 
you get to put it all together. (White female WES student) 

R2: Right, because you try to explain things to people and even if you say something 
wrong, usually they'll know if it's wrong, you know, I don't know, it just seems to work 
that way, so I mean, you learn something from them, and they learn something from you, 
so and I guess I think the real learning is, yeah, from us all talking in groups. I mean, if 
we had to sit there and work by ourselves on these worksheets, I don't think it would be 
very fun or helpful. (White female WES student) 

Rl: I don't think I'd finish one problem, personally. 

R3: I'll agree. I mean, I think it definitely comes from explaining it to other people 
because then you gain the confidence that you know and that's a huge part of it, is just 
knowing that you know it instead of like just well, I'm not sure if I really can explain this 
to someone else, but once you explain it to them and you see that they understand it then 
you know that you have an accurate understanding of the material. (White male WES 
student) 



*** [music]*** 

Rl: I guess it has pleased me in a way to learn that, that it's okay to pick up the phone 
and call someone and it doesn't mean that you're dumb or whatever. I mean, I guess the 
reason why whenever I call someone to ask them about the homework that I feel bad is 
because, there was someone in one of my brother's engineering classes who is always 
calling him and asking him, Dan, have you done the homework set yet, you know, and 
Dan would be like yeah, and so he'd spend a lot of time helping this person. But his 
impression of the person who was calling was that they hadn't done any of the work on 
their own and I don't want anybody to have that impression of me,... (Hispanic-American 
female WES student) 

I: Has WES had an impact on that feeling, I mean, how is that? 

Rl: Yeah, I guess because I'm seeing other students around me who I know who work 
very hard who still have questions and do call other people and network, and so I guess, 
that's been helpful... When you have questions piling up, I would never feel like I could 
call someone right away or ask. I had to save that question and ask the professor, 
because the professor was the only person who was okay to ask. But it's so much easier 
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just when you have the question... to solve it right then and there, instead of saving the 
question and forgetting about it. 

I: So solve it right then and there by doing what? What, 

Rl: By picking up the phone and calling the TA or calling a fellow student. 

R2: My sister is taking 221 right now and she doesn't, I mean, she doesn't really have 
friends from discussion. I mean, no one that she can call and ask, and I mean, usually I 
can help her, but they, you know, are approaching the material a little bit differently but 
sometimes I can't, and you know, she's been sitting really frustrated like, "I can't get this." 
And I'd never really had to deal with that because I always felt comfortable like calling 
these guys and saying, "You know, I don't get this." There's like three or four people I 
can call and if not I just go right down my WES list of people's numbers and you know, I 
know people well enough to be sure that they're not going to be like, you know, upset 
about me calling ... I've been frustrated working problems but I always knew that there 
was kind of an out for me, whereas she doesn't have that, so it kind of makes me 
appreciate WES even more because, I mean, I've made such good friends there and people 
I can call and you know, they know where I'm coming from. They know how I think 
mathematically and I know how they t hink mathematically, so we can really bounce 
thin gs off one another. (White female WES student) 

***[music]*** 

Baine: These excerpts convey a number of interrelated reasons why the group work process 
fosters both interdependence and independence. The student assistants believe that the group 
work approach fosters independence because " you really understand when you explain it 
yourself, " and group work forces students "to explain it for themselves. " 

Susan: The idea that group work fosters independence was also central to what students in 
the first focus group said: as one student put it, "You gain confidence when you explain it to 
other people because once you see that they understand, then you know you have an accurate 
understanding of the material." The students also explained that with group work, a person 
not only learns when she explains something to the others, but also learns when others in the 
group take their turn explaining things. 

Baine: Now, one might ask why having another student explain something would be different 
from having a professor explain it. The student interviews suggest that it's different because of 
the way they relate to their peers. For one thing, learning through group work gives them 
confidence because they discover they are on an equal footing with their peers. As one 
student put it, seeing other students who she knows work very hard and who "still have 
questions" makes it "okay" that she also has questions. She doesn't feel intimidated by her 
hardworking and smart peers— although she still feels awed by her lecture professor. 
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Susan: For another thing, they establish a kind of mutual understanding which students 
rarely share with professors. Each student "knows how the others think mathematically, so 
they can really bounce things off one another. " 

Baine: Another explanation the students gave for why group work helps them learn is that it's 
more immediate to be able to get help from one another. They don't have to "wait to talk to 
a professor to solve the problem. " 

Susan: Overall, group work for these WES students fosters mutual interdependence-as each 
member both contributes his or her differing skills and knowledge and seeks help and 
affirmation from the others, and also fosters a sense of self-reliance. 

***[music]*** 



[2. Worksheets] 

Susan: Consider again the idea that there are three critical factors associated with the 
functioning of the WES Program— group work, worksheets, and instructors— which are 
interactive and can be visualized as three overlapping circles. 

Baine: We've presented typical student perspectives about how and why the group work 
element works, and will now consider the second main learning process factor-the 
worksheets. In doing so, please remember that the worksheet factor interacts with the group 
work factor in ways that are essential to the functioning of the program as a whole. 

Susan: Recall that WES worksheets are comprised of difficult problems which the WES 
instructors produce for each workshop session. To convey the role these worksheets play, we 
present three excerpts that focus specifically on the worksheets. The first and second are 
from student focus group interviews, and the third is from an individual interview. 

***[music]*** 

I: How would you say the real learning takes place in the WES workshops? 

SI: I think when he gives us a problem that isn't like, like when you're doing the 
homework, I've done this a lot of times, do the homework and you say, okay, a line 
intersects a plane, you flip back two pages and you look at the equation that says a line 
intersects a plane at, and then you fill in the numbers. And on the WES worksheet he'll 
say that, but say it in a way that you can't exactly just flip back into the book and copy 
the equation. So you actually have to figure it out and decide what exactly the cross 
product is doing and why it's there and that's, I think, where stuff starts to make more 
sense instead of being just a bunch of numbers. (White female WES student) 

*** [music]*** 
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Rl: You open up a book, you read it and you do problems, that's not real learning. You 
feel like that's kind of like mechanics, you kind of do it, blah, blah, blah, blah. It's kind 
of like step-by-step. You look back in the section and say, "Oh, plug in these numbers." 
That's not real learning, that's more like practicing. Learning has been, he throws us a lot 
of theoretical stuff, and the worksheets take off of that, so that we don't just sit in lecture 
and be like a robot going through. We actually go to discussion and pull it apart piece by 
piece and do it ourselves all the way again and you actually kind of like start to 
understand the whole concept and stuff like that. I mean, it's slow but I think going 
through and doing what he did in lecture ourselves is learning because then we're not just 
watching it and writing it down, we're coming up with it ourselves too. And then you're 
like, "Oh! Well that's where the formulas come from." That's the difference. ...(White 
female WES student) 

I: Can you describe that process? I mean, is this on your own or as a group, 

Rl: No, on the worksheet in WES, she'll say, "Recall from lecture that he.." and then 
she'll write down an equation that we know personally it took him like 40 minutes to get 
to, you know, and then she'll say, "Well, do it, start from the.." And we're like, "No 
way! It took him so long!" But then we'll go through, well get out our little rulers, well 
draw the picture that he kept drawing over and over and well label this angle, that angle, 
and well derive each part of the picture until we come up with a final formula at the end. 
And that's learning, I think— knowing how to get it all put together instead of opening a 
page and saying there's the formula. 



***[music]*** 

R: ...being a freshman, I just came out of high school and here I come to a class. I'm 
expecting my college discussion courses just to be sitting in a classroom or a TA being at 
the board, whereas this discussion is pretty much you walk in, you sit down with your 
group, and they hand out worksheets and you do the worksheets together as a group. This 
was back in high school again, you know, "Here is a worksheet, you must do this before 
you leave," and it seemed kind of too structured. But to tell you the truth, it's much more 
productive because of the social aspect, as well as the organization of the worksheets 
themselves. The questions in the worksheets tend to be even more difficult than the ones 
in the book and tend to be specialized toward the lecture itself, where the book tends to 
be more general. (African-American male WES student) 

I: And you like that? 

R: Hm hmm I like the creativity of the worksheets, because of that creativity it makes 
you think in more general ways. I mean, it tends to make you think more. The book 
kin d of tells you how to do it and then gives you problems in order to use their way of 
thinkin g. Where the worksheets tend to encourage even more ways of thinking. 



I: Hmm , I don't understand that. I mean, are the problems... 

R: Well, the problems generally have more words as in, more English in them, where the 
book generally has equations or says, you know, "Solve for this using this," or whatever. 
And the English will make you think of how to answer the problem. And as well, the 
professor, at the beginning of the hour, will sometimes make comments about the 
worksheets which also will make you think. 

*** [music]*** 

Susan: The key to the worksheets, as these students explained it, is that they make the 
students "actually have to figure it out. " As one put it, "we do it ourselves all the way so that 
we 're coming up with it ourselves. " Moreover, the worksheets tend to encourage "more 
creative ways of thinking, " taking them beyond just "plugging numbers into formulas. " 

Rainer We learned from our interviews that the difficult and carefully chosen worksheet 
problems not only encourage students to be more creative and "figure it out themselves, " but 
also motivate them to rise to a challenge. 



[3. Instructors - There "When and Only When" You Need Them] 

Susan: Now let's turn to the last of the three learning process factors that are critical to the 
functioning of the WES Program. We've presented the group work element and the worksheet 
element. The last is the instructor factor. The WES instructors are effective because they 
are, as we like to put it, "There for the students when and only when they need them. " 

Baine: This idea that the WES instructors successfully help the students learn calculus 
because they are there "when and only when" the students need them helps explain why, when 
we reviewed all the transcripts from our interviews with students, we found relatively few 
comments about the instructors. While not particularly visible in the interviews, the 
instructor element is no less essential than the other two elements. 

Susan: 1 would like to stress that the following two excerpts-both of which are from student 
focus group interviews~are entirely representative of students' attitudes toward the WES 
instructors, as apparent not only in our interviews but also from our many observations of the 
WES classrooms. 



***[music]*** 

R: ...I'm very in awe of the fact that calculus is one of my easier classes. And I think 
that is because I spend those six hours in WES doing extra problems and I don't even 
know how I would study if I were in a regular discussion because obviously I would need 
more time. I know I would need to do more problems, but where would I find them? I 
don't know if they would give us more in discussion. I don't think so because they just go 
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over homework. So I think it's in having extra problems and having six hours to get really 
expert, useful help on those problems right there as soon as you don't understand it, 
they're there to explain why. Where if you're stuck on a problem and you go the next day 
to your TA's office hours, you're like, "Well, I was doing this but I don't really remember 
what I didn't get or what I was doing," and that sort of thing, so I think it really helps a 
lot to have someone right there to help. (White female WES student) 

***[music]*** 

Rl: Before our exam this past, I think it was this past exam, our group had no idea what 
was going on. Denise explained it to us, and then after she went over it and everything, 
and it wasn't so much as telling us what to do, but she more explained it. It was more us 
asking questions of a certain thing instead of her exactly telling us. I think I learned more 
that two hours than in any other two hours... (Hispanic-American female WES student) 

***[music]*** 

[B. Outcomes Associated with the Interaction of the Three Process Factors] 

Susan: Having presented students views on what we consider the three key learning process 
factors, we turn now to consider more fully how these three factors interact. As we've noted 
throughout, we imagine these three factors-the group work, the worksheets, and the 
instructors-as overlapping circles. It is apparent to us that the interaction of the three 
factors is essential to the effectiveness of the WES program. 

Baine: We believe that the interaction of these factors produces a learning outcome which, 
while not easily assessed in calculus exams, is an outcome which most math faculty value the 
most for their calculus courses. To be specific, we believe that the combination of the group 
work approach, the worksheets, and instructors who take the "guide on the side" role fosters 
in students the interest-and ability-to acquire a deeper, more conceptual understanding of 
calculus. 

Susan: Our next four excerpts convey the idea that the WES students, routinely confronted 
with challenging problems through the worksheets, and supported and encouraged by their 
peers and instructors develop persistence and a capacity to use multiple ways to solve 
problems. The first and third excerpts feature various WES instructors, and the second and 
fourth excerpts feature WES students. 



***[music]*** 

Melinda: People have documented throughout school that women get better grades. 
They'll do what you tell them, you know, and they're very conscientious. But they also 
take it harder if they don't do as well. They don't seem to want to keep going, you know. 
If they're [not?] getting it pretty nearly perfect, then "Forget it, I'll do something else." 
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I: And the WES experience, your intention 



Melinda: I'm saying it will counteract that because so many of the problems in the 
workshop, nobody can get them right away. So you're used to seeing a problem there that 
looks unsolvable, but then you tackle it and you see that it begins to yield after a while, 
and even if you don't get it perfectly, you know, there's a sense of satisfaction, so I think 
it counteracts this feeling that, this sort of all or nothing. 

*** [music]*** 

I: If you look back over your time in WES, can you think of a time when you were 
particularly pleased or especially pleased with your work? 

Rl: I can tell you ...our project is to come up with a problem and present it to the class. 
And there was a problem on a worksheet and it was like the last one. You never really 
get to the last ones because they are the ultimate, forget them, they're out of the question. 
And I said, "Hey, I ought to do this for my problem." She goes, "Oh, go ahead." 

.. people had tried but nothing was working. She's like, "I think you're going to have to 
take a different attempt." And I decided to start this last night because I have to do this 
on Tuesday, tomorrow, and I'm working, and working and working and working. I'm like, 
"I'm never going to get," but I have to at least say I've tried before I get her help because 
shell be like, "Oh, you haven't done anything." So I worked and worked and worked and 
worked for hours and hours. I just didn't give up, and finally, I'm just working along 
thinkin g this is another dead end and I got the answer! I think that is the happiest and the 
proudest of myself this entire year I've felt and I was just, "I got it! I got it! " I was like 
ecstatic. It would be like getting a raise or something, ...I just know it's right, and I 
figured it out and didn't ask a single question along the way. (White female WES student) 

I: Now tell me, back at the beginning of WES, how would you have approached it? The 
same way or, 

Rl: In the beginning, I remember those worksheets and I'm like, "What! Are they crazy? 
There's just no way we have the ability to make it through these insanely hard, abstract, 
absurd problems." I would have said, "Oh, well, there's just no way." And I don't think I 
would have believed in myself enough to have gotten through it. So my approach 
probably wouldn't have been to keep at it for that long because I would have been like, 
"Oh, I'm never going to get this, I'm wasting my time. I should be doing more productive 
things." But last night I'm like, "Maybe I could get this, maybe if I keep going I'll get 
this." So, that's the difference between beginning and now--is just because going through 
so many hard problems, I know that it's in my ability. 

***[music]*** 

I: ... would describe from your perspective what the WES approach to teaching is, for 
you? 
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Dave: I think it's very much letting the students learn the material instead of forcing it on 
them. I know my own experience learning second semester calculus, the material was 
very much forced on me, I never figured anything out for myself. You know, here's the 
material, here's a proof, here are three examples, go do 15 more just like that one. There 
was never, you know, how would one figure this out, how would one approach this 
problem? ...That's one of the big advantages--it shows students more of the real side of 
mathematics and much less this thinly-veiled, very application-oriented, can you do the 
brute force problems. 

I: I don't really understand what you mean by the real side of mathematics. 

Denise: "Why as opposed to how," I think is a good explanation. 

Dave: Yeah. If you can see three examples and do 15 more just like them, you certainly 
know how to do the problem, you may have absolutely no clue why, why are you doing it 
the way you are. You know, questions arise all the time in class— "Why am I crossing 
these two vectors?" And WES students seem very willing to ask these questions, and 
eager to ask these questions, and unfulfilled until they've answered them themselves. 

*** [music]*** 

R: ...another thing that I experience in all my classes is that I think now that I've finally 
gotten into college I'm like thinking different about things, and I can understand the stuff 
deep behind calculus, you know. Like first semester it was well I just want to know how 
to do it, you know, so I can do it on the exam. But I think this semester WES has helped 
to understand how it works and what's behind it and why. (White female WES student) 

*** [music]*** 

Susan: This completes our presentation of the research findings we developed on the basis of 
interviews with students and instructors and classroom observations. As I hope is clear, we 
have come up with an expanded notion of course outcomes for the students in the WES 
program based on three interactive learning process factors. To wit, we found that the 
program encouraged students to: 

develop higher levels of confidence in their mathematical abilities; 
value multiple and more creative ways of problem-solving; and 

develop the interest and the ability to acquire a deeper, more conceptual understanding of 

calculus. 



***[music]*** 
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[V. Quantitative Analysis of WES/TD: Interview with Professor Joel Levin] 

Susan: Having made this analysis on the basis of our qualitative research data, many 
listeners still may wonder if these apparent improvements in the learning process really 
translate into improvements in achievement as measured on standard calculus exams. Do the 
apparent process improvements translate into statistically significant improvements in GPA 
and course completion? Because we also think these are very important questions, I asked 
Joel Levin-who, as noted before, is a UW-Madison professor of Educational Psychology-to 
perform a statistical analysis of the available data on this 1 993-94 pilot WES program. In 
the next portion of our audio program he describes his analysis at some length. 

*** [music]*** 

I: I have to say that I was very pleased that you agreed to work with us on this evaluation 
project. One of the things that I was wondering is if you had any comment to make about 
what approach you took to analyzing the student data on the Emerging Scholars Program? 

Joel: Sure. The first thing to realize is that this is not what one would call a rigorous 
research experiment or even an intervention, an educational intervention. To do that, one 
would have had a number of things done differently than was in place for this particular 
demonstration, one might call it. In a real research evaluation, one would have to have 
had some flexibility over the assigning of the different students to the different conditions. 
And the conditions being the WES program and the traditional discussion section. 

I: Right, it is not like you worked with the professors ahead of time to say that this is the 
way the students should be assigned. 

Joel: The way that is typically done in educational research or through more scientific 
research, as you know, is through random assignment to the different sections. That 
wasn't possible here. So we had a group that was already available to us. So what we had 
to do in evaluating the effects of the program were to take into account as much relevant 
information that was available to us and to account for that or adjust for that statistically 
in coming up with any kind of conclusion about the program. So we had to make due 
with what we had in this particular case. 

In terms of the statistical analyses, we also did not have a good situation in that the 
groups were meeting as a group and we don't have a condition that is called 
"independence," that is necessary to perform some standard statistical techniques. So 
alternative techniques had to be applied where that was possible to do. So we did two 
kinds of analyses. One much more liberal; one much more rigorous. 

I: Which one was more liberal? 

Joel: The liberal analyses we are calling the "student level analyses," where all the 
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students in the program get compared with all the students who were not taking the WES 
program—where we did not worry about the fact that people were meeting as a group and 
interacting as a group. We just analyzed individual students' grades in the course. 

I: Now why do you say that is more liberal? 

Joel: It is more liberal because if there is something effective going on in the section that 
is potentially influencing everyone in the section, it is not independency demonstrating 
that whatever intervention you are trying is effective. If there is something that is going 
on that is effecting everyone, those are not independent of one another within a section. 

To do valid statisical comparisons, a key assumption is that the units that are being 
analyzed—the individual scores within each section~is independent of one another. That 
can be done in laboratory research where you test people; you administer treatments 
individually and independently. But in a classroom or a group situation, you are not going 
to be able to have that independence unless you have multiple sections. So the individual 
level analyses are more liberal because they are presumably, and it would be easy to 
demonstrate, almost certainly violating the statistical assumption of independence. 

So a more conservative approach is what is called a "section-level" analysis, where instead 
of inputting individual scores into the analyses, use the average of each small section as 
the scores or the grades that will be put in. So instead of putting in fifteen or twenty 
grades in the analysis, one would just put in the average of each of the WES discussion 
sections. And so in those analyses there would be two WES discussion sections being 
compared with either 64 non-WES sections in the 221 course, or 43 non-WES section 
averages in the 222 course. That is viewed as a more appropriate and more conservative 
test of the effectiveness of the program. 

I: Well, what did you find? 

Joel: Well, either way there is a definite effect. When one compares the results from the 
two approaches, the conclusions are very, very similar. For the person who is less 
rigorous, they would focus on the individual-level analyses. For the person who wants to 
be more careful and cautious, they would focus on the section-level analyses. In both 
cases, there was an impressive difference between the WES sections and non- WES 
sections, and one cannot criticize the more conservative approach taken because it was 
done using the sections as units of analysis. 

One controls for the variables that one thinks are related to achievement in these math 
courses and we found that two variables were important to take into consideration. Other 
variables were also, but once these two variables were identified, the other variables did 
not add anything to that. The two variables were how they did in their high school 
classes, their rank in their high school classes. And how they did on the math placement 
test, the gamma test at the university, which indicates what their prior math skills and 
background are. 
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I: And that is a test that has been designed by the University of Wisconsin? 

Joel: Right. There are actually three or four of these standardized tests that are used. But 
the gamma test, the thir d level test, is the one that we used. That together with high 
school rank is a very good predictor, a combined predictor of what they are going to do in 
these courses. So we took those into account when comparing the WES students and the 
non- WES students. So what is referred to as "statistically adjusting" are controlling for 
those measures when comparing their grades in the math courses. So it is sort of like 
saying, "How would these two groups of students compare in their math grades when we 
have tried to equate as much as possible for these two predictors, these two variables?" 

I: So that you could sort of say that at least at a statistical level, they all look pretty much 
the same as they go into these courses. 

Joel: Right, right. Statistically they have been equated. 

I: And one is getting the regular discussion section treatment, and the other is getting the 
Emerging Scholar treatment. 

Joel: But again, we are being cautious. Another level of control we used in being 
cautious here is we created two groups to compare the WES students with. We compared 
the WES students not just with students who were not in WES, but among those who 
were not in WES, we created those who had the same large lecture instructors and those 
who did not. Because one could argue that maybe the reason that WES students might 
have done better is that the lecture instructor they have got was better. 

I: So you wanted to look at that factor. 

Joel: So that would be a way to control for instructor differences. There were other 
sections that were taught by the same instructor, but did not get the WES discussion 
group. So when we looked at that, we found no difference at all in the achievement 
between two non- WES sections. 

And whether analyzed at the student level or at the section level, the WES students 
statistically outperformed both of those comparison groups. The magnitude of the effect 
was about a half of the grade, .5 grade point in the 221 class. So, for example, they were 
averaging better than a B average in the WES section at the student level. And the non- 
WES students were averaging at about C+, B/C level. So it was half of the grade, the 
difference between a B/C and a B on the average, 2.5 versus 3.0. The difference increased 
even a little bit more to six or seven tenths of the grade point in 222 sections. I have 
much more confidence of the validity, the utility of the program, even more so in the 222 
manifestation of it. When one looks at the section level analyses, for example in 222, the 
two WES sections out of 45 sections came out 1 and 2 with respect of their grade point 
average. They were top two sections out of 45. 
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I: The question of whether the student drops the course or not. Did you take into account? 

Joel: We did take that into account. What we did was a couple of different analyses. One 
we just did an analysis of those who completed the course. That is, they got a grade- 
anything other than a drop or a withdrawal or an incomplete. And there were higher 
percentages. There were hundred percent completion rates among the WES students in 
222. There were equally high or higher completion rates between WES and non-WES 
students in the 221 course. So even if one says how did they compare when you control 
for drops and withdrawals, etc., the WES came out higher. 

A second way that we analyzed was to use a very, very conservative estimate of what 
they would have gotten in the course. 

I: That is if they hadn't dropped. 

Joel: If they hadn't dropped, we assumed that perhaps some of them would have gotten 
low grades. Most of the ones that dropped would have gotten low grades. So if one 
assigns them essentially a D- or between a D and a F, and includes them in the analyses, 
the same picture comes out. So even incorporating them into analysis, giving them a D- 
average, one comes to the same conclusion. The WES students, for whatever reason, the 
WES section is the higher, performs statistically higher then the non-WES section. 

I: As you know, the way is which this program was originally developed by the people at 
Berkeley and Austin and so forth, there was a special focus on whether or not the 
Emerging Scholar methodology, had an especially strong positive effect for under- 
represented students in mathematics. In particular, they were looking at ethnic minorities. 

I think here in Wisconsin people are also interested in the effect of the program on 
women. I am wondering if you saw any effects that were interesting with respect to the 
differential performance of women and minorities in Emerging Scholars? 

Joel: To break them down by the different ethnic groups was something that we tried to 
do, but the numbers were not there to allow any kind of definitive conclusions. 

I: Too few. 

Joel: Too few, right. When you break them down by three of four ethnic categories, the 
pattern looks similar. One can't talk about them statistically, but there was nothing unusual 
when one looked at the profiles across the three different comparison groups. For women 
versus males, again, the patterns that were seen at the overall level were pretty much 
replicated for the males and females. 

I: Okay. So what you are saying that overall students of all types performed better in 
WES? 
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Joel: At this point. Given the limited numbers of students we have and the limited ability 
to break them down that way. So on the other hand, one can't say that it is particularly 
beneficial for one group over another. But on the other hand, one can say that it seems to 
be comparably beneficial for all groups. 

I: Do you have any other comments that you would like to make from your standpoints as 
a person who has come at this study from a statistical orientation? 

Joel: The only thing I would say is that I certainly went into this trying to be as impartial 
as I could about what was going to be found here. And the numbers were going to have 
to prove themselves to me. And I subjected these data to quite a test. We discovered 
that there was a potential that students who started right after their high school graduation 
and started immediately in the fall are in a different category than those who take Math 
221 after coming through the university and taking it either after having failed it before or 
after having taken preparatory math courses. So we had to control for that also in some 
analyses, which we did. So there were a number of things we discovered along the way. I 
was really trying to say "show me" or "prove it to me" that this is going to stand up even 
when one takes these other variables into account. And I must say, I am quite convinced 
statistically that the group that got this instruction certainly are different in their 
achievement in those courses, 221 and 222, at the section level than the group that didn't. 

I: One of the things that we are hoping to do as this program develops, is to get a sense 
of the likelihood of success in calculus over time for students of different types. So in 
other words, take all the women who start calculus in the fall of their freshmen year. How 
many of them will have been retained to their intended math-based major as opposed to 
the men who started in a similar way? How many of them will have been retained? At 
these point we don't actually have those data. But I am wondering if we are going to be 
able to see any effect there. 

Joel: It would be very interesting. As we talked about before, to be scientifically credible, 
one would have to adopt a different kind of research methodology and a tactic in 
designing this program so that one could make more generalizable, definitive statements 
about whether WES is solely responsible for these differences or are there other things? 

And then the other thing would be to see how WES students do in other courses, whether 
these are transferable, generalizable skills that they are learning that will help them not 
just in other math classes, but have formed a basis with which they can succeed in 
college. 

I: And do you think we could look at that from a statistical standpoint? 

Joel: Oh, certainly. If there were enough common courses that these students took or to 
just follow them through the first year. If what we are seeing in the first year of the 
program predicts how they persevere in math courses and math careers. The intent of 
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Treisman was to introduce students from under-represented groups to continue in math- 
related careers. Well that is what one could obviously see here. 

I: Since we have only had the '93 - '94 pilot cohort to look at, it is a little early. But we 
are hoping to get a chance to look at those things. 

Joel: And I do think that the quality of these data and the kinds of inspections that have 
been made of them, at least equal if not surpass what have been done in this area in the 
past. At least from the Treisman reports and results that we have seen. And it would be 
very interesting and hopefully you will try to disseminate some of these results and see 
what other people at other institutions think of them. 

I: Well, we are very much looking forward to doing that. Thank you, Joel. 

Joel: You are welcome. 



*** [music]*** 



[VI. Implementation Problems] 

Baine: The pilot WES Program, like any pilot program, encountered some problems. We 
focus on these during this last portion of our audio program. Both the students and 
instructors described various difficulties which we believe are a function of implementing the 
program. In other words, our interview data support the idea that these are 
"implementation" problems rather than flaws in how the program itself is designed. As the 
interview excerpts in this section of our program indicate, both students and instructors 
encountered some difficulty in enacting the processes that seem to make this program work. 

***[music]*** 



[A. Worksheets] 

Susan: We heard many students explain that they need the first one or two problems on the 
worksheets to be easy enough that they can get warmed up, or "get into gear. " 

Baine: We also heard them explain that they want the problems to be difficult enough that 
the group is forced to work together to solve them, but shouldn't be so difficult that they can't 
solve them even when they work on them as a group. To illustrate this point, we present an 
excerpt from an interview with an individual WES student. 

*** [music]*** 

R: Sometimes I get frustrated on the worksheets because I'm stuck on number 1 and and 
I'm getting a little frustrated because I want to get this worksheet done. So it might be 
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nice to have like maybe two that kind of just get you up to speed, kind of similar to ones 
out of the book. Because even, you know, even the hard ones in the book to me are 
never as hard as what Professor Bleicher puts on his worksheets, and I appreciate having 
the hard stuff to work on but I'd like to have something that maybe gives me a little 
confidence to get going, like I know what I'm doing, and then finish the worksheet with 
the kind of stuff that he has. But those are really the only revisions that I'd make. (White 
female WES student) 



***[music]*** 



[B. WES Isn’t for Everyone] 

Baine: A few students expressed frustration about the attitudes of some of their fellow WES 
students. These students felt that each student who is part of the WES program had to be 
serious about the program. At issue here is that same interdependence we discussed earlieR: 
because group work depends on working together, every member of a group has to hold up 
his or her part of the bargain. 

Susan: In some sense, then, this is a selection issue. Students who are not going to take the 
WES program seriously should not elect to participate in it. This point is made quite clearly 
by students in the following two excerpts. The first excerpt is from an individual interview 
and second is from a focus group interview. 

***[music]*** 

I: Do you think it would work for everyone? 

R: I'd say no because there's a girl in our class who goes to, a discussion section and then 
comes and.. .sits in on our discussion. For some reason, she says she feels that she studies 
better alone, then why join WES, that makes no sense. (African-American male WES 
student) 

I: Was she in WES from the beginning? 

R: No, she dropped in this semester. 

I: Okay. 

R: So I have no idea. So it might not work for everybody. If people aren't good in 
groups they should not be in that class because if somebody's not good in a group then 
they're going to, if you're working on a problem and you thi n k you're just going to speed 
ahead and not wait for anybody else, then you shouldn't actually be in that class at all. 

I: Do you think people learn to work in groups? 




129 



28 



R: ...I worked in groups in labs and all that other stuff, you know, but I guess, I'd say for 
me I didn't learn. But maybe a couple people have learned to work in groups in that 
class. It's not that hard to learn to work in a group. It all comes down to if you can be 
patient with another person or if you can respect their views and opinions and accept what 
they're saying or else if you can combine what you're saying and what they're saying 
together. That's what it comes down to when you're working in a group. 

I: Okay, do you think everyone does feel free to speak their mind in that class? 

R: To an extent, some people are still shy to the point where they're thinking like, "Well, 

I really don't want to look stupid in front of these people because I know that person over 
there knows what they're talking and I really don't so I don't want to say anything that can 
make, me sounds dumb." ...but, like when somebody asks a question like that it might not 
come out the first time clear in what they're saying to the person. But if you pick up on 
what they're saying then you resay again, you know, the answer might come out of the 
problem because I mean everybody has intelligence and nobody's stupid. 

***[music]*** 

I: Are there any things about WES that aren't so great? 

Rl: It's really frustrating to be with people who don't seem like they really want to be 
there and I'm sure that will change as the program gets more organized and they know 
who is best in there and you know, what kind of people should join. But it's very 
frustrating that, you know, we'll all be working really hard and we'll get it and then people 
will show up at the review session and they'll be like, "Um, what's a vector?" and we're 
like, "Shut up, go away." That's very frustrating, that we're split, or you know, Professor 
Bleicher's time and Dave's time is split between us, who are trying to understand it, and 
this other person who is trying to get an A on the exam or just trying to get the two 
credits or whatever that is. (White female WES student) 

R2: I think that's a big deal. A lot of people don't really have a desire or don't really 
even like calculus, but are in WES and I think they'll get weeded out as this program 
continues to grow. But that's very frustrating to see people right behind you come in late 
and not really care and you kind of have, kind of almost a team concept to this t h ing and 
it's like, any thin g like a team in sports, like when one person is a weak link in the chain, 
the chain breaks and it's kind of frustrating sometimes. But you just try to sometimes 
keep a closed c hain to the people that have a desire and try to work hard with that I 
guess. (White male WES student) 



***[music]*** 
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[C. Instructor Issues] 

Susan: The WES instructors explained that the most difficult problem they encountered in 
implementing the WES program was learning how to change how they interact with students. 
In short, it was difficult for them to shift from their lecture mode to the guide on the side 
mode. Central to this shift was learning how to strike the right balance between challenging 
students with difficult problems that they have to solve on their own, and giving enough 
direction so they don't grind to a halt with frustration. 

Baine: The first excerpt is from an interview with the undergraduate student assistants. The 
second is from an interview with the faculty who, as we noted earlier, acted as WES teaching 
assistant during the pilot year of the WES program. 

***[music]*** * 

Dave: I think one of the most important things that I found was seeing when not to 
approach a group when they're working reasonably well. And even though they're stuck 
on a point in a problem, not telling them how to go on because eventually they probably 
will figure it out, especially in some groups. And then also seeing when they're so stuck 
on a problem that they really need more of the background material in order to figure it 
out. 

I: It sounds like you're sort of playing their level of frustration, watching when they fall 
over the edge of excessive frustration. 

Dave: Yeah, and in some sense playing my own level of frustration because it's 
frustrating to see students not getting problems. You want everybody to be able to do all 
the problems and the easy solution is to go show them how to do the problem, which is 
certainly not in the spirit of the program and not in the spirit of this style of teaching. 

I: So it's a question of self-restraint almost. 

Dave: Hm hmm, definitely. 



*** [music]*** 

Mike: Yeah, the other thing in that regard is we've been teaching a long time. Maybe 
Melinda's not as habit-bound as I am, but it's very hard to change from your.. .lecture 
mode to "give them a few hints now and then" mode. 

Melinda: It is very hard. 

I: I wonder if you'd elaborate on that a little by giving an example of where you, say, 
found yourself slipping between what you would call the WES approach and ...the 
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approach that you're more accustomed to. 



Mike: Well, it's hard to come up with a particular example. But just, like a student might 
be asked to find say an n-vector analysis part of the, find the equation of a plane 
perpendicular through some line through some point, you know. That's a fairly stock 
question. And if they're stuck and they say, "Well how do you do this?" it's very easy to 
say, "Well, you have the normal vector to the plane. You should be able to write down a 
generic equation of a plane with that normal vector and then find out what the constant 
has to be to mak-p. it go through this point," which is what someone would normally do, 
rather than say, "Well let's go back to the plane. How would you do it if it was a line 
and a point and you wanted to have a line perpendicular to a given line through a point?"- 
-sort of makp. them think it through themselves in three dimensions. The impulse is to 
say, "Well, you know, take these steps." And that's what you don't want to do. You want 
to make them think of the steps. 

Melinda: I definitely found that the hardest part. The other thing is that it was hard for 
me not to want to alleviate their misery when they were stuck. I think it will be easier 
for teachers who haven't, aren't quite as experienced. 

Mike: ...bound to the old system. 

Melinda: Yeah, I found that hard and I also found if the very first problem on the 
worksheet that everyone in the room was having trouble, ...I occasionally— and I still don't 
know if this is a good thing to do--I had the tendency occasionally to give them a little 
hint as a group and that becomes teaching. 

I: Hm hmm, so you're sliding back into 

Melinda: Yeah, definitely. I think it's a feeling of control too, you know. You feel the 
class sort of slipping away from you. 

I: I'm interested in that. I wish you would pursue that. 

Melinda: I think this is a hard thing for a teacher who feels a certain responsibility to 
establish the personality of the whole class... you bring to it your enthusiasm for your 
subject hopefiilly, and you want to co mmuni cate that to them, and you want the whole 
class to sort of absorb it from you. In the old way of teaching, I sort of felt like, it was 
up to me, not that it's totally possible for any teacher to do this— but to establish the 
feeling of the class. 

I: Now by implication you're saying that in WES you don't do that? 

Melinda: In WES you don't do that, no you definitely don't do that. 
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I: Now what does that mean? Do you have to give up something? 

Melinda: I think you do. I think as a teacher you do give up that feeling of control, for 
sure. 

I: Okay, and that's uncomfortable for you? 

Melinda: That's uncomfortable, that is hard. That is very hard. I think 

Mike: You're unused to it, because in some sense you still feel a full responsibility, I 
mean, if these students don't do well, you feel you know, it's something I've done, you 
still feel the responsibility. 

Melinda: Yeah, I agree. 

Mike: To helping your students learn the material and do well in the course and have 
successful careers. You don't want to be the responsible party for them not being 
successful. And yet the method of teaching this way is such that they have to do it 
themselves... You sort of force them to do things by not doing it, and yet when you see 
them not doing what you feel they need to be doing, there's a very strong tendency, "Oh, 
they're getting off the track and I better straighten them out." 

Melinda: Yeah, there is a point at which, of course, you can't let them just get totally 
demoralized with not a clue how to do any problem. So I think it's a question of balance, 
but I certainly think you have to get used to a feeling of not being in control of the class 
and this is really hard to get used to. 

I: Okay, what I'm hearing you say, Melinda, is there's a question of letting go, of almost 
trusting them. 

Melinda: Yeah, exactly... In fact, I remember Treisman saying something like this, that 
you have to sort of trust the mathematics...you have to trust the power of mathematics to, 
I mean, that sounds a little mystical, but 

\ 

Mike: And the force will be with you. 

Melinda: Right, something like that. 

I: Well does it work? 



Melinda: It does work, definitely it works. 

I: The force is with them? -» ^ ~ 
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M elin da: Yeah, because mathematics is very powerful. It works, but it's a question of 
degree, of course. But it's just that you do have to tolerate a higher degree of chaos. 

Mike: Yeah, the classroom is definitely chaotic. If someone walks in you've got all these 
groups of students laughing and joking and working on their mathematics, but not in any 
reasonable semblance of order in the traditional classroom sense. They may be going off 
on paths that you know are not going to lead them to solutions, and yet you sort of have 
to bite your tongue and say well, eventually they'll realize that that's not going to get them 
a solution and that they better try a different approach. 

Melinda: It's very hard though. 

Mike: The pedagogue, you know, the traditional pedagogue as probably Melinda and I are 
Melinda: Probably, definitely. 

Mike: We want to say stop, go back and start again, but it's a mistake to do that before 
they realize they've made a mistake. 



***[music]*** 

[D. The Group Work Approach Is Effective but Actual Work Groups Vary in Character 
and Effectiveness] 

Susan: A related implementation problem that all the WES instructors and a few students 
noted pertained to establishing work groups that functioned well AS groups. These speakers 
make clear in their comments a pattern which we noted when observing the WES workshops: 
some work groups were more effective as learning and social communities than others. 

Baine: One important difference among the groups was that some functioned as loose 
coalitions of people who mostly worked on their own and checked with each other from time 
to time, while others worked together on a continuous basis. 

Susan: In these last three interview excerpts, we hear various individuals present their views 
on problems they encountered while implementing the work group approach. First, we hear 
from the two undergraduate student assistants. We then hear from an individual student. And 
last, we hear from Mike and Melinda, the faculty "TAs. " 

***[ music]*** 

Dave: I think one of the big problems with the WES program, at least in how it was run 
this year, was not paying attention to group dynamics enough. 

I: Could you give examples? 
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Dave: Sure. There are excellent groups who work really well together. In Bleicher's class, 
the group which is in the center table... They work wonderfully together, they're always 
arguing, they're always doing the problems. And then you have at the end of the room a 
more quiet scene, and they were very rarely arguing, mostly working problems on their 
own, comparing answers. The group dynamic was nothing near what the other groups and 
in that case they're much more apt to ask you a question and just have you answer it for 
them, and then they say, "Okay, I understand," and go back to working the problem on 
their own. Whereas if you come over and help the table in the middle, they'll all listen, 
all try to understand, and as soon as one person understands you can leave and you know 
they'll explain it to the rest of the group, you know. It's a very much a group attitude. I 
think we lost a significant number of students— at least lost their attention— because the 
group they were working in was not an effective one. 

I: When you say "lost," you mean lost them to the full possible benefit of the method? I 
mean, because they stayed in the class. 

Dave: We lost them in terms of they were then more apt to go to the problems on their 
own and think that that was the way you had to learn the material, and I think that was a 
big problem. 



***[music]*** 

I: What would you say, as far as you personally are concerned, would be the most 
frustrating elements about not only your WES experience, but your classes in general? 

R: The most frustrating thing I find is when you're supposed to be in a team environment 
and there is no team playing going on, as in you have four people and three are in a 
group and one is left by himself or by herself, or there's a two and two, the fact that this 
is supposed to be a group effort yet someone or some people are left out. (African- 
American WES male) 

I: Now does that happen in some of your other classes or does it happen 

R: It happens in WES, it does happen in WES, and in WES it's even more irritating 
because it's supposed to be a team effort. That's the main emphasis and when you're kind 
of left out of the group, and especially when you're not understanding it and you need 
someone else's point of view, or someone else to help you with it, and they're not, that 
gets very frustrating and that will tend to make you just forget about it, kind of toss it to 
the side. My other classes are the same way, but because it's not an emphasis so much on 
team work, it's not so much of an irritation. Because you kind of go, "Well, I kind of 
expect this, I'll do it on my own anyway," it's not that much of a problem. 

I: Because you don't expect it anyway. 
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R: Hm hmm. 



I: You're working on your own. 

R: For the most part, yeah. 

I: Has there been any way in which you've seen the faculty member or the undergraduate 
senior peer help address that problem that you described? 

R: That particular problem, I believe, I can't remember any incidents that either the TAs 
or the professor actually did help in situations. It's more of a "social thing," you know, 
little groups amongst themselves. 

I: Well what do you do? 

R: In that particular situation, 111 try to, kind of force my way into it. 

I: Force your way into? 

R: Well, for instance, if there's a group of four individuals and I'm not particularly 
understanding something and the other three people seem to understand but they're not too 
open... then 111, you know, make it more obvious. And if that doesn't work, then 111 
probably go the TAs or the professor and try to talk to them to make sure I understand it. 

I: I see. Would you wander into another group? 

R: The way the WES Program is set up, once the groups have been set at the beginning 
of the class, those are the groups you stay in. There's no real trading back and forth or 
anything like that, so you're pretty much stuck with the group you're in. 

I: So your recourse is to go to one of the faculty 

R: Hm hmm. If you can't go, if you can't talk to the rest of the group, then you are 
forced to talk to the professor or the senior TA. 

*** [music]*** 

Mike: Well there are some things I would organize differently in my class... things that I 
wish I had known before I started this, and that was the way to organize the class and the 
whole problem of whether you form groups or whether you form them, students, let them 
form their own groups. 

Melinda: That was a very major question. 
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Mike: Yeah, I was much too lax I think letting them form their own groups and not, 
probably once at least every other week they ought to be sort of juggled around a little 
bit. 

Melinda: Well, at least several times a semester. I would certainly say there are probably 
several things we could both say looking back at the year of experience that we would 
love to advise the next group to do. 

I: Why don't you go ahead and do that? 

Melinda: Well, there are two main components, actually in a way, that who is in what 
group and the problems themselves. In some sense the success depends almost 
exclusively, well aside from the teacher and the student assistant which are other 
important components, but, I think, assuming that those leaders are reasonably good I 
think that the make up of the group and the nature of the problems themselves are the 
crucial components. Actually once I got into it and we had a lot of resources, thanks to 
Mike having culled many problem sets from other programs, I felt I kind of got the hang 
of that. But the groups and the, you know, we're not experienced group psychologists or 
anything, in a normal class you don't even know any of these things are going on, so my 
first recommendation would be to echo Mike, I would definitely change the groups at 
certain intervals, I don't know if every other week, but certainly I would do that. 

I: And when you change them would you let them go back to their groups of choice in 
between your times of choosing? 

Melinda: Good question, I have no idea. 

Mike: I think I might... there were one or two dysfunctional groups, well one really in my 
session, that I should have much earlier just broken it up and not let it re-form. I might 
just have them pick cards or pick numbers out of a hat or something. 

***[music]*** 

Baine: What's interesting is you have these instructors getting used to the "guide on the side" 
role and learning that part of this role is that they have to actively attend to how the groups 
are functioning, and intervene when they are not functioning well. 

Susan: Thus, while they avoid intervening in the group process when students are actually 
doing worksheet problems, they must not avoid intervening when it comes to structuring of 
the work groups and helping students learn how to work in groups. 

***[music]*** 
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Bailie: This concludes our audiotape program on the University of Wisconsin-Madison's pilot 
year of the Emerging Scholars Program. We hope it was useful for you, and would urge 
listeners who would like more detailed information about the pilot WES Program to consult 
the audiotape cassette case for our address. 

Susan: Last, since our business is evaluation, we would very much appreciate hearing your 
evaluation of this audiotape program. We invite you to check the cassette case for 
information about how to call, E-mail or write us with your comments. Thank you for 
listening. 
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